One node surprisingly running out of disk space

Question

I had two cardano-node instances running via kvm on one physical machine. Both nodes were running on mainnet for a while before I switched them over to testnet (to prepare for turning these into a stake pool).

I gave both environments (labelled BlockProducer and Relay) the same resources, incl. 150 GB of disk space.

When logging into both environments, I noticed that BlockProducer's cardano-node process crashed with a FileSystemError, stating that there is no space left on the device. Restarting the process (also after rebooting BlockProducer) did not change this.

These are the outputs for df -h and du -h -d 1 in the respective cardano-node directories. Left is the (working) Relay, while the right side shows the BlockProducer environment.

The host system running the kvm daemon still has 210 GB of available disk space. Did I simply under-provision the virtual disks or what is going on here?

Why is Relay still running though? I installed the same software and configured both environments the same.

Answer 1

1. Disk space issue:

Each system only has 73GB maximum on / (which is where you are running from). The live Cardano chain is now 50GB on it's own. I'm not sure of the testdb size.

Make the / volume larger on each system eg 120GB maybe.

2. Differences between BP and Relay system:

The BP system has used 70GB of / while the Relay has used 65GB of /

So, what has used the extra 5GB somewhere on the BP system? Left over logs or code?

cd to /home and do a 'du -h .' there to see if you can spot if the space has been used by a different user.

If not obvious there then you will need to review other directories off of / to narrow down where the extra 5GB has been used. Often /var/log can use a lot of disk space logging things. Also check /tmp

Answer 2

You need to find out which files are filling up the disk.

A command like:

find / -type f -printf "%15s    %p\n" | sort -nr | head -30;

might help.