1

I recently created a new server and install of Cardano node and db sync (1.34.1 & 12.0.2) on same machine. I implemented the snapshot restore to bring the Postgres DB up to EPOCH 325 and was able to successfully connect and query the data without issue. However, now I would like for DB sync to start loading any new (delta) data beyond EPOCH 325, but the DB sync appears to be "sitting" at an operation called "Deleting 8751 blocks up to slot 55342638". The operation has been running for 10+ hours and I'm starting to become concerned that the DB sync is not working.

NOTE I do understand that the DB sync operation can take a considerable amount of time, especially for the more recent blocks. However, from a data processing perspective this is getting hard to comprehend, especially without being able to see logs or events that could show progress.

I'm sure others are experiencing the same issue and if you were able to find a solution, please do share. All help is very much appreciated.

Improve this question

5 Answers 5

Reset to default
2

I am the main dev on db-sync.

"Deleting 8751 blocks up to slot 55342638"

The message that is to be expected right after that is "Blocks deleted", but deleting 8751 blocks should only take 10s of minutes at most providing the machine you are running on meets the minimum system requirements.

Improve this answer
3
  • Hello, thank you for the support! Yes, the machine has been configured to exceed the requirements you referenced. It would be great if there was a way to trace the events to help diagnose the issue. As the main dev, could you please help me in producing the necessary logs so that I can troubleshoot this issue? Thank you again.
    – Nicholas
    Commented Mar 26, 2022 at 16:36
  • If there is nothing in the db-sync logs, then this is a PostgreSQL issue.
    – Erik de Castro Lopo
    Commented Mar 27, 2022 at 3:45
  • Thank you Erik. PostgreSQL logging it is then! I'll be sure to update once I have this issue identified.
    – Nicholas
    Commented Mar 28, 2022 at 13:00
1

I have the same issue. Because of corrupted connection to a remote cardano-node. After connection got fixed, db-sync still wants to remove 203 blocks (the other day 213) and that takes forever.

Mar 31 14:13:46 ip-10-31-12-145 cardano-db-sync[682]: [db-sync-node:Info:59] [2022-03-31 14:13:46.35 UTC] Deleting 203 blocks up to slot 57168135

Actually, I have never seen db-sync to finish that procedure and I was always switching to manual snapshot restore procedure, that is more predictable in that case.

My machine is beefy AWS instance running dbsync and postgres v14 only.

I would appreciate any Postgres troubleshooting tips.

db-sync logs show nothing after message about deletion. In processes I see:

postgres 776 663 98 14:13 ? 00:17:00 postgres: 14/main: cardano mainnet [local] DELETE

Improve this answer
1
  • Hello, Thanks so much for commenting on this thread. My db sync node has been operating without issue after completing the block deletion. I continue to research Postgres DB logging / profile trace methods, but "work" keeps getting in the way. I'm sorry I cannot provide further support, but I will be sure to update once I have established the appropriate DB profile tracing to help troubleshoot issues such as this.
    – Nicholas
    Commented Apr 7, 2022 at 4:04
0

Apparently, the DB sync process just takes a really long time to "initiate". Not sure if that is the right term, but I will use it for now.

I just checked the db-sync status and it has completed the deletion operation and has moved on to processing the deltas.

I believe that the db-sync utility is an absolute must in the Cardano Ecosystem and I look forward to the opportunity to further improve this technology.

Happy exploring everyone!

Improve this answer
0

Took about 2 hours for me to delete 318 Blocks... My Read IOPS is a bit low (133K Read / 44.5K Write), 2 x 500GB SSDs Raid1, Intel Xeon-E 2136 - 6c/12t - 3.3 GHz/4.5 GHz

2022-06-07T02:21:37.182748120Z [db-sync-node:Info:57] [2022-06-07 02:21:37.18 UTC] Rolling back to slot 62991407, hash b3adef1cb2d8f642f51d60212a719f92080f94b3455c8001e211bf244fbbac3b
2022-06-07T02:21:37.184475177Z [db-sync-node:Info:57] [2022-06-07 02:21:37.18 UTC] Deleting 318 blocks up to slot 62998495
2022-06-07T02:21:37.193243543Z [db-sync-node.Subscription:Notice:47] [2022-06-07 02:21:37.19 UTC] Identity Required subscriptions started
2022-06-07T04:02:32.178760338Z [db-sync-node:Info:57] [2022-06-07 04:02:32.17 UTC] Blocks deleted
2022-06-07T04:03:31.899791964Z [db-sync-node:Info:57] [2022-06-07 04:03:31.89 UTC] Found snapshot file for slot 62991407, hash b3adef1cb2d8f642f51d60212a719f92080f94b3455c8001e211bf244fbbac3b
2022-06-07T04:03:44.397586843Z [db-sync-node:Info:60] [2022-06-07 04:03:44.39 UTC] getHistoryInterpreter: acquired
Improve this answer
3
  • 1
    > 133K Read / 44.5K Write That's almost unmanageable for large database processing, you'd expect your instance to lag a lot
    – RdLrT
    Commented Jun 7, 2022 at 5:19
  • Yeah, I've been planning on upgrading the memory, but good to know I need to upgrade the storage as well
    – yungtechboy1
    Commented Jun 7, 2022 at 23:06
  • Sure, just to be sure , you might want to double-check system requirements for dbsync if you havent - those are predominantly aimed for mainnet configuration
    – RdLrT
    Commented Jun 8, 2022 at 6:18
0

This was caused by some missing indexes. cardano-db-sync 13.x.x has already these indexes set-up.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.