I have read all those links too and tried many different things. I still get some missed slots.
The things I have found:
-I0 (Default garbage collector with idle garbage collection off):
With this set I don't get missed slots at all for many hours and then eventually the garbage collector has to run and copy a lot of memory and then I get many missed slots all at once.
-I0.3 -Iw600 (Default garbage collector with regular collector sweeps with minimum 10mins between sweeps when processor idle > 0.3 secs)
This results in few missed slots which occur when the garbage collector runs. However, because it is done more regularly the amount of memory copying is less and it takes less time so there are less missed slots each time. But for my setup there were still a couple of missed slots every 10 mins when the GC ran.
--nonmoving-gc
I still get occasional missed slots.
Memory use seems to gradually increase over time and the node eventually uses all available RAM 16G and starts using swap. After running 20hrs rts_gc_peak_megabytes_allocated is 24G which seems excessive. Node appears to be running OK with almost no missed slots (4 in 20hrs).
If you run a small pool with limited processor power/speed then one option is to do the following:
Work out when your allocated slots are by running cncli-leaderlog ahead of time. Then plan to restart your block producer cardano-node service 20 minutes before you are due to produce your block. This way you can be sure your memory is fresh and garbage collection won't be necessary for a while just at the time when you are due to produce your block.
Overall, throwing faster hardware at the problem is an easy fix. Faster processor means memory copying is quicker and garbage collection takes less time so missed slots are eliminated.
Update 2021-12-21:
After lots of testing I have concluded that the only RTS option which stops my node from missing slot leader checks is --nonmoving-gc. However the problem with this setting is that memory is never released back to the OS with GHC version 8.10.7. Unfortunately we need to wait until the cardano-node is able to be compiled using GHC version later than 9.2.x
What I currently use is:
+RTS -N --nonmoving-gc -RTS
As I understand things, the other settings compiled in (set by IOG in the source code) are also enabled. These include: "-I0 -A16m". You can check what your binary has with:
cardano-node +RTS --info
The output for my version currently is:
("Flag -with-rtsopts", "-T -I0 -A16m -N2 --disable-delayed-os-memory-return")
Then I restart cardano-node every few days at a convenient time between block production.
Or check your memory usage periodically with:
free
And restart cardano-node when you get close to exhausting your RAM, or after this but before too much swap is used.
If you just let cardano-node keep running using the nonmoving-gc then eventually all the RAM and swap is used and the OS kills the process using its out of memory reaper. I believe this is because GHC version 8.10.7 doesn't return the collected memory to the OS.
Addition 2021-12-21:
OK now I am not sure about the last bit. Memory is eventually returned to the OS when using nonmoving-gc. Here are some outputs of 'free' while running continuously with:
'+RTS -C0 -N -I0 --nonmoving-gc -RTS'
Tue 21 Dec 2021 10:04:57
total used free shared buff/cache available
Mem: 16393504 15851756 161420 20 380328 262232
Swap: 17039352 13271360 3767992
Tue 21 Dec 2021 22:02:44
total used free shared buff/cache available
Mem: 16393504 15325260 648972 20 419272 786832
Swap: 17039352 11186532 5852820
Note that 12hrs has elapsed and RAM and Swap usage has reduced a little.