Missed slot leader checks

Question

The “missed slot leader checks” on my stakepool are at 0.98% or 14,527. Is this cause for concern? I have tried applying the solutions found in the following links, but nothing seems to make a difference, links →

• https://forum.cardano.org/t/too-many-missed-slots-happens-help-me/75942/44
• http://www.htgsd.com/cryptocurrency/ada-cardano-stakepool/missed-slot-leader-checks-missed-blocks-with-stakepool-cardano-node-ada/
• https://forum.cardano.org/t/solving-the-cardano-node-huge-memory-usage-done/67032/37
• https://forum.cardano.org/t/no-more-missed-slots-during-epoch-after-changing-one-setting/67924/20

My BP is running on a Contabo VM (VPS M), 6 cores, 16GB RAM, 400GB SSD hard drive and 400 Mbit/s, hosted in Germany. My 2 nodes are running on two VMs, with specifications 4 vCPU cores, 8GB RAM with a 10GB swap setup, 200GB SSD hard drive and 200 Mbit/s. My stake pool was prepared following the CoinCashew tutorial.

Does anyone know what I am doing wrong? Thanks so much, Simon

Answer 1

I have read all those links too and tried many different things. I still get some missed slots.

The things I have found:

-I0 (Default garbage collector with idle garbage collection off):

With this set I don't get missed slots at all for many hours and then eventually the garbage collector has to run and copy a lot of memory and then I get many missed slots all at once.

-I0.3 -Iw600 (Default garbage collector with regular collector sweeps with minimum 10mins between sweeps when processor idle > 0.3 secs)

This results in few missed slots which occur when the garbage collector runs. However, because it is done more regularly the amount of memory copying is less and it takes less time so there are less missed slots each time. But for my setup there were still a couple of missed slots every 10 mins when the GC ran.

--nonmoving-gc

I still get occasional missed slots. Memory use seems to gradually increase over time and the node eventually uses all available RAM 16G and starts using swap. After running 20hrs rts_gc_peak_megabytes_allocated is 24G which seems excessive. Node appears to be running OK with almost no missed slots (4 in 20hrs).

If you run a small pool with limited processor power/speed then one option is to do the following:

Work out when your allocated slots are by running cncli-leaderlog ahead of time. Then plan to restart your block producer cardano-node service 20 minutes before you are due to produce your block. This way you can be sure your memory is fresh and garbage collection won't be necessary for a while just at the time when you are due to produce your block.

Overall, throwing faster hardware at the problem is an easy fix. Faster processor means memory copying is quicker and garbage collection takes less time so missed slots are eliminated.

Update 2021-12-21:

After lots of testing I have concluded that the only RTS option which stops my node from missing slot leader checks is --nonmoving-gc. However the problem with this setting is that memory is never released back to the OS with GHC version 8.10.7. Unfortunately we need to wait until the cardano-node is able to be compiled using GHC version later than 9.2.x

What I currently use is:

+RTS -N --nonmoving-gc -RTS

As I understand things, the other settings compiled in (set by IOG in the source code) are also enabled. These include: "-I0 -A16m". You can check what your binary has with:

cardano-node +RTS --info

The output for my version currently is:

("Flag -with-rtsopts", "-T -I0 -A16m -N2 --disable-delayed-os-memory-return")

Then I restart cardano-node every few days at a convenient time between block production.

Or check your memory usage periodically with:

free

And restart cardano-node when you get close to exhausting your RAM, or after this but before too much swap is used.

If you just let cardano-node keep running using the nonmoving-gc then eventually all the RAM and swap is used and the OS kills the process using its out of memory reaper. I believe this is because GHC version 8.10.7 doesn't return the collected memory to the OS.

Addition 2021-12-21:

OK now I am not sure about the last bit. Memory is eventually returned to the OS when using nonmoving-gc. Here are some outputs of 'free' while running continuously with:
'+RTS -C0 -N -I0 --nonmoving-gc -RTS'

Tue 21 Dec 2021 10:04:57

               total        used        free      shared  buff/cache   available
Mem:        16393504    15851756      161420          20      380328      262232
Swap:       17039352    13271360     3767992

Tue 21 Dec 2021 22:02:44

               total        used        free      shared  buff/cache   available
Mem:        16393504    15325260      648972          20      419272      786832
Swap:       17039352    11186532     5852820

Note that 12hrs has elapsed and RAM and Swap usage has reduced a little.

Answer 2

Your relays are under the recommended memory specifications that is current 12GB of RAM for cardano-node. This might or might not be the root cause.

Answer 3

I am sorry to tell you this, but from what I saw (other people complaining about the same thing), the Block Producers hosted at Contabo have a lot of missed slot leader checks. This is most likely because of the over-provisioned cpu cores. Missed slot leader checks are happening when the CPU is busy doing other tasks and does not have enough time to check each slot (each second) if it should mint a block or not. Also from my experience on my own block producer on mainnet (because on testnet I do not have any missed slots, even on epochs transitions), using any other RTS option than -N will not improve this metric. Using all the CPU cores is the only option that really helps here. Most of the missed slot leader checks happen when the cardano-node service is doing a snapshot of the ledger on the disk. This happens once every hour (if I'm not mistaking) by default. You can change this behavior by adding the "SnapshotInterval" setting in your mainnet-config.json file and setting a larger value. for example:

  "SnapshotInterval": 86400,

for one snapshot every 24 hours. This will reduce a lot the missed slot leader checks. The downside is probably a longer start time of the service, but from my observation and calculations, it is a matter of a few extra seconds, probably under 10 seconds, so it is negligible. I am using this in my config, exactly as above, and I only have 2 missed slot leader checks every 24 hours. Before adding it, I had 2 every hour, so it is obvious what made the difference. But my BP has 8 CPU Cores and 24 GB RAM, on Contabo you will still probably have some missed slot leader checks, but this should decrease. If it does not decrease a lot, you might want to find a new hosting for your Block Producer, if you thing it might cause you missed blocks.

Answer 4

You miss slot leader check when CPU is busy with something else. Remember slot is 1 sec so each sec (slot) the BP needs to check if its leader. Now you always know when you forge the block before epoch but that is not the case for the node. It has to check because there could be slot battles.

So is your Contabo VM having dedicated CPU? the check has to finish within that sec else the slot is missed. Is this a problem? if you are not forging block then not that much. But in general its always best to not have too many missed slot except during the epoch end which is a known problem. Also note that if you are using GHC options which trigger garbage collection etc then sometimes and more regularly you will miss slots. restart and Watch for 24 hours and see when the first slots are missed and how regular they are. There is no good number for missed slots but main thing is you should not miss the slot when you are forging block

Answer 5

RAM 16GB may not be sufficient when the network is busy. If +RTS ... -RTS doesn't help, you can try setting "TraceMempool": false in the mainnet-config.json file. This will give you some extra free space in RAM.