1.35.3 not syncing since epoch 365

Question

I moved our stake pool LOXE to 1.35.3 (git rev 80b5637a5520648d50b0763d7677ddbe374cd598) on August 20th and it was producing blocks regularly. I noticed yesterday that our block producer was up as a slot leader, but the block was not adopted by the chain. Only later did I realize that our pool was stuck in Alonzo at the last block of epoch 364 (7791698).

I stopped the nodes and deleted some recent files from the db, fo force replaying of blocks on restart with no difference.

I complied cardano-node and cardano-cli - this time from commit ea6d78c7 - transferred the binaries and configs to my nodes and compiled the libsodium on the nodes as well. I changed the topology on the relays to just connect to the IOHK servers for the moment to try and just sync up, but I’m still getting nowhere with it.

Attached is a view from my journalctl from the relay in question.

UPDATE: Fixed!

Will compose and link a write-up

Answer 1

The disconnect came from gLiveView reporting the version number of the cardano-node that it found, not the one that's actually running.

Here's a more complete answer: https://forum.cardano.org/t/1-35-3-stuck-in-epoch-364/108111/22

Answer 2

I think what I am about to say is correct but check it out. If you are talking about a mainnet pool you are running the wrong node version. The correct git rev for mainnet is 950c4e222086fed5ca53564e642434ce9307b0b9. The node version you are running is for the test environments.

To get up and running quickly, when you're on the correct node version you could download a db snapshot from here:

https://csnapshots.io/

Answer 3

For those getting stuck with updating fully to 1.35.x. This guide assumes that your user directory is named cardano, your git directory is in cardano/git, your working directory is in $NODE_HOME, and your systemd service is cardano-node.

Check your installed cardano-node version

cardano-node --version

now compare if it is the same as being executed by systemd

which cardano-node  
/home/cardano/.local/bin/cardano-node --version

Assuming you're not sure where your 1.35.x file is located you are finding it by running:

find ~/ -name cardano-node

This list will also list directories with the name cardano-node.

Look for a path similar to this, where [YOUR_GIT] is the cloned repository.

/home/cardano/git/dist-newstyle/build/x86_64-linux/ghc-8.10.7/cardano-node-1.35.3/x/cardano-node/build/cardano-node/cardano-node

Now test this node's version

/home/cardano/git/dist-newstyle/build/x86_64-linux/ghc-8.10.7/cardano-node-1.35.3/x/cardano-node/build/cardano-node/cardano-node --version

Copy this cardano-node if the version is 1.35.x

cp /home/cardano/git/cardano-node/dist-newstyle/build/x86_64-linux/ghc-8.10.7/cardano-node-1.35.3/x/cardano-node/build/cardano-node/cardano-node /home/cardano/.local/bin/cardano-node

and

cp /home/cardano/git/cardano-node/dist-newstyle/build/x86_64-linux/ghc-8.10.7/cardano-node-1.35.3/x/cardano-node/build/cardano-node/cardano-node /usr/local/bin/cardano-node

Limit the permissions

sudo chmod 644 /etc/systemd/system/cardano-node.service

Check version

/home/cardano/.local/bin/cardano-node --version /usr/local/bin/cardano-node --version

Reboot sudo reboot

or reload the daemon

sudo systemctl daemon-reload
sudo systemctl enable cardano-node.service

If it doesn't run now.

Stop the cardano-node.

sudo system stop cardano-node

Backup your old database.

mv db db_bak

Download your snapshot from cSnapshots.io and reboot.

sudo curl -o - https://downloads.csnapshots.io/mainnet/$(curl -s https://downloads.csnapshots.io/mainnet/mainnet-db-snapshot.json| jq -r .[].file_name ) | lz4 -c -d - | tar -x -C $NODE_HOME; sudo reboot

Your node should now hopefully be running again and you can delete the backup with

rm -r $NODE_HOME/db_bak