Since the current network settings define one slot per second and each block that reaches a node was created for a specific slot, the local node can determine how "on time" the block arrived.
Let's first define a common understanding of the block distribution process before we get into the meaning of these new metrics:
Each block-generating node, when elected as a leader for a slot, has a time budget of one second to validate the accumulated transactions and scripts, generate the block, and broadcast it.
This is usually done via its relays, which also validate the block, adopt it, and then re-offer it to its peers for download.
After 2 to possibly several hops, the block will reach the block creator destined as leader for one of the upcoming slots. This can already be the next second, but on average it happens every 20 seconds according to current parameters.
Each node in this delivery chain will try to complete its part of the work as quickly as possible. Larger blocks with multiple transactions mean more work and therefore longer validation and transmission times. In addition, there are certain phases in each epoch where each node performs additional tasks in the background, such as the epoch switch itself and the reward calculation which currently starts 48 hours after each epoch switch and is then performed in smaller sub-steps for about 18-22 hours. This background work currently also has a noticeable influence on the processing and distribution times of the blocks, as can be seen in the following example graphics.
All these new metrics are derived from the first time a new block for a given slot reached the node.
_blockdelay_s
is the delay between for which slot the last block received was produced and when it actually reached the local node (without the additional time required to validate it localy). This value will get updated the next time a new block for a new slot reaches the local node.
_lateblocks
counting blocks that arrived more than 5s after they where supposed to have been forged. Ideally the block should arrive within the same slot = second. Larger values mean a certain risk for the block producer to not see the very last produced block in time, and build their own block based on the second last produced block, which means the following producer and whole chain may have to solve a fork. So this metric idealy should always remain at 0. A relative high growth compared to other nodes means this node has some issues to get the blocks in time.
_blockdelay_cdfOne _blockdelay_cdfThree _blockdelay_cdfFive
These three metrics are averages based on the blocks received in the last 6 hours. So they also have a corresponding creeping progression.
Shown are how many blocks arrived within one, three and five seconds. Ideally, all blocks should arrive within one second = slot. You can see in the following graphics that this is largely the case, but especially in the reward calculation window it drops very clearly, because of the background activities mentioned above.
The three-second value is less affected by this. And once again significantly more blocks reach the node within 5 seconds. For the security specification of the Ouroboros Praos protocol, this 5-second value is the important and critical one. It is important that the vast majority of all blocks (95%) have reached every other block creator in the global network within 3 or max 5 seconds.
Some example graphs
The first graph shows _blockdelay_cdfOne for 3 different nodes, together with the processed block sizes (pink) in a timerange of 20 days = 5 epochs.
85-95% of all blocks arrive within one second.
The drop 48h after each epoch switch because of the reward calculation load is clearly visible, lowering the receive-within-1sec-rate to 5-15%
Also visible the relation of large-block phases delaying the block propagation and lowering the 1sec-rate.
The following graph shows _blockdelay_cdfThree and _blockdelay_cdfFive for the same 20-day time range:
We see only a slight effect on the 3-second and almost no effect on the 5-second metric.
The following graph zooms in to a couple of minutes, and shows the direct relation of individual block sizes (pink), and the resulting _blockdelay_s
Last but not least the new node metrics also show cardano_node_metrics_forks_int
We see a slowly but steadily growing number of forks. A natural and expected thing in Ouroboros proability base decentralised slot leader election. But we also see the growth rate increasing in the reward-calculation window, which means this is caused by extended propagation delays and certain blocks produced without having received the very last block in time.
Update 2022-01-11
Since pools started upgrading to node version 1.33, the previously described 48h window for the reward calculation becomes re-scheduled to cover almost the whole epoch. This will result in less impact on the 48h window.
The following graphs shows this together with a second effect we currently see reducing the CDF1 ratio: more large blocks.
At the moment of this writing, ~ 64% of all nodes was updated to v 1.33
