Why my cardano-node is now running a strange loop after an upgrade to a higher version?

Question

Lately, after a node upgrade to a higher version (1.30.1), my Block-Producing (BP) node will "stay" in the "start" phase for about 45 minutes making a strange loop by switching to synchronization mode and then immediately going back to the "start" phase.

[2021-09-30 03:17:34.14 UTC] Replayed block: slot SlotNo 40283991 of At (SlotNo 40321032)
[2021-09-30 03:17:37.33 UTC] block replay progress (%) = 100.0
[2021-09-30 03:17:37.72 UTC] before next, messages elided = 40284006
[2021-09-30 03:17:37.72 UTC] Replayed block: slot SlotNo 40305522 of At (SlotNo 40321032)
[2021-09-30 03:17:37.72 UTC] Replayed block: slot SlotNo 40305529 of At (SlotNo 40321032)
[2021-09-30 03:17:40.52 UTC] block replay progress (%) = 100.0
[2021-09-30 03:17:40.58 UTC] before next, messages elided = 40305616
[2021-09-30 03:17:40.58 UTC] Replayed block: slot SlotNo 40321032 of At (SlotNo 40321032)
[2021-09-30 03:17:40.58 UTC] Opened lgr db

I'm running my node on a 8GB RAM Ubuntu LTS with 4 vCPUs. Any thought anyone?

Answer 1

Some SPOs in the community have reported that since version 1.29.x of cardano-node it requires at least 10GB of RAM to function properly. If your node has 8GB of RAM, maybe you should try adding swap (2GB or 4GB).

Assuming that you don't have any swap partition configured yet (Check it by running this command, if there is nothing back at the output, it means you currently don't have any swap space).

sudo swapon --show

STEP 1 - Create your swap file

sudo fallocate -l 2G /swapfile

Verify that the correct amount of space was reserved:

ls -lh /swapfile

you should see:

-rw-r--r-- 1 root root 2.0G Sep 30 22:59 /swapfile

STEP 2 - Enable the swap file

sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Verify that the swap is now available

sudo swapon --show

the output should be:

NAME      TYPE  SIZE USED PRIO
/swapfile file 2048M   0B   -2

STEP 3 - Add it to fstab to make it permanent at startup

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab