1

I moved our stake pool LOXE to 1.35.3 (git rev 80b5637a5520648d50b0763d7677ddbe374cd598) on August 20th and it was producing blocks regularly. I noticed yesterday that our block producer was up as a slot leader, but the block was not adopted by the chain. Only later did I realize that our pool was stuck in Alonzo at the last block of epoch 364 (7791698).

I stopped the nodes and deleted some recent files from the db, fo force replaying of blocks on restart with no difference.

I complied cardano-node and cardano-cli - this time from commit ea6d78c7 - transferred the binaries and configs to my nodes and compiled the libsodium on the nodes as well. I changed the topology on the relays to just connect to the IOHK servers for the moment to try and just sync up, but I’m still getting nowhere with it.

Attached is a view from my journalctl from the relay in question. journalctl for cardano-node

UPDATE: Fixed!

Will compose and link a write-up

4
  • You say you upgraded the block producer machine, but did you also upgrade the relays? They must all be upgraded. Also, you should use the git tag rather than just using a particular commit near the tag. Sep 25, 2022 at 4:22
  • I upgraded all nodes.
    – Matthias Sieber
    Sep 25, 2022 at 5:21
  • To be of help I would need a bit more info. Are you running in P2P mode, how many incoming/outgoing connections for your block producer and relays? Was the block you were leader for accepted in gLiveView but not by other pools? Sep 25, 2022 at 13:50
  • Connections between BP and relays were stable. Connections to other nodes are impermanent and break soon after they were registered in gLiveView. I don't have any indication that the block was accepted by other pools as the tip is on the las slot in epoch 364.
    – Matthias Sieber
    Sep 25, 2022 at 16:28

3 Answers 3

1

The disconnect came from gLiveView reporting the version number of the cardano-node that it found, not the one that's actually running.

Here's a more complete answer: https://forum.cardano.org/t/1-35-3-stuck-in-epoch-364/108111/22

0

I think what I am about to say is correct but check it out. If you are talking about a mainnet pool you are running the wrong node version. The correct git rev for mainnet is 950c4e222086fed5ca53564e642434ce9307b0b9. The node version you are running is for the test environments.

To get up and running quickly, when you're on the correct node version you could download a db snapshot from here:

https://csnapshots.io/

0

For those getting stuck with updating fully to 1.35.x. This guide assumes that your user directory is named cardano, your git directory is in cardano/git, your working directory is in $NODE_HOME, and your systemd service is cardano-node.

Check your installed cardano-node version

cardano-node --version

now compare if it is the same as being executed by systemd

which cardano-node  
/home/cardano/.local/bin/cardano-node --version

Assuming you're not sure where your 1.35.x file is located you are finding it by running:

find ~/ -name cardano-node

This list will also list directories with the name cardano-node.

Look for a path similar to this, where [YOUR_GIT] is the cloned repository.

/home/cardano/git/dist-newstyle/build/x86_64-linux/ghc-8.10.7/cardano-node-1.35.3/x/cardano-node/build/cardano-node/cardano-node

Now test this node's version

/home/cardano/git/dist-newstyle/build/x86_64-linux/ghc-8.10.7/cardano-node-1.35.3/x/cardano-node/build/cardano-node/cardano-node --version

Copy this cardano-node if the version is 1.35.x

cp /home/cardano/git/cardano-node/dist-newstyle/build/x86_64-linux/ghc-8.10.7/cardano-node-1.35.3/x/cardano-node/build/cardano-node/cardano-node /home/cardano/.local/bin/cardano-node

and

cp /home/cardano/git/cardano-node/dist-newstyle/build/x86_64-linux/ghc-8.10.7/cardano-node-1.35.3/x/cardano-node/build/cardano-node/cardano-node /usr/local/bin/cardano-node

Limit the permissions

sudo chmod 644 /etc/systemd/system/cardano-node.service

Check version

/home/cardano/.local/bin/cardano-node --version /usr/local/bin/cardano-node --version

Reboot sudo reboot

or reload the daemon

sudo systemctl daemon-reload
sudo systemctl enable cardano-node.service

If it doesn't run now.

Stop the cardano-node.

sudo system stop cardano-node

Backup your old database.

mv db db_bak

Download your snapshot from cSnapshots.io and reboot.

sudo curl -o - https://downloads.csnapshots.io/mainnet/$(curl -s https://downloads.csnapshots.io/mainnet/mainnet-db-snapshot.json| jq -r .[].file_name ) | lz4 -c -d - | tar -x -C $NODE_HOME; sudo reboot

Your node should now hopefully be running again and you can delete the backup with

rm -r $NODE_HOME/db_bak

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.