5

Searched docs.cardano.org, github, forum.cardano.org and the rest of the internet for some type of documentation that describes each of the items in the EKG metrics, but couldn't find anything. I understand about half the metrics, but there are some new ones with version 1.31.0 of whose meaning I don't know. Would definitely like to know what these mean

cardano_node_metrics_blockfetchclient_lateblocks cardano_node_metrics_blockfetchclient_blockdelay_s cardano_node_metrics_blockfetchclient_blockdelay_cdfOne cardano_node_metrics_blockfetchclient_blockdelay_cdfFive

1
  • Wow, I would really like to see that documentation! Dec 10, 2021 at 19:02

1 Answer 1

9

Since the current network settings define one slot per second and each block that reaches a node was created for a specific slot, the local node can determine how "on time" the block arrived.

Let's first define a common understanding of the block distribution process before we get into the meaning of these new metrics:

Each block-generating node, when elected as a leader for a slot, has a time budget of one second to validate the accumulated transactions and scripts, generate the block, and broadcast it.

This is usually done via its relays, which also validate the block, adopt it, and then re-offer it to its peers for download.

After 2 to possibly several hops, the block will reach the block creator destined as leader for one of the upcoming slots. This can already be the next second, but on average it happens every 20 seconds according to current parameters.

Each node in this delivery chain will try to complete its part of the work as quickly as possible. Larger blocks with multiple transactions mean more work and therefore longer validation and transmission times. In addition, there are certain phases in each epoch where each node performs additional tasks in the background, such as the epoch switch itself and the reward calculation which currently starts 48 hours after each epoch switch and is then performed in smaller sub-steps for about 18-22 hours. This background work currently also has a noticeable influence on the processing and distribution times of the blocks, as can be seen in the following example graphics.

All these new metrics are derived from the first time a new block for a given slot reached the node.

_blockdelay_s is the delay between for which slot the last block received was produced and when it actually reached the local node (without the additional time required to validate it localy). This value will get updated the next time a new block for a new slot reaches the local node.

_lateblocks counting blocks that arrived more than 5s after they where supposed to have been forged. Ideally the block should arrive within the same slot = second. Larger values mean a certain risk for the block producer to not see the very last produced block in time, and build their own block based on the second last produced block, which means the following producer and whole chain may have to solve a fork. So this metric idealy should always remain at 0. A relative high growth compared to other nodes means this node has some issues to get the blocks in time.

_blockdelay_cdfOne _blockdelay_cdfThree _blockdelay_cdfFive These three metrics are averages based on the blocks received in the last 6 hours. So they also have a corresponding creeping progression.

Shown are how many blocks arrived within one, three and five seconds. Ideally, all blocks should arrive within one second = slot. You can see in the following graphics that this is largely the case, but especially in the reward calculation window it drops very clearly, because of the background activities mentioned above. The three-second value is less affected by this. And once again significantly more blocks reach the node within 5 seconds. For the security specification of the Ouroboros Praos protocol, this 5-second value is the important and critical one. It is important that the vast majority of all blocks (95%) have reached every other block creator in the global network within 3 or max 5 seconds.

Some example graphs

The first graph shows _blockdelay_cdfOne for 3 different nodes, together with the processed block sizes (pink) in a timerange of 20 days = 5 epochs. 85-95% of all blocks arrive within one second.

enter image description here

The drop 48h after each epoch switch because of the reward calculation load is clearly visible, lowering the receive-within-1sec-rate to 5-15% Also visible the relation of large-block phases delaying the block propagation and lowering the 1sec-rate.

The following graph shows _blockdelay_cdfThree and _blockdelay_cdfFive for the same 20-day time range:

enter image description here

We see only a slight effect on the 3-second and almost no effect on the 5-second metric.

The following graph zooms in to a couple of minutes, and shows the direct relation of individual block sizes (pink), and the resulting _blockdelay_s

enter image description here

Last but not least the new node metrics also show cardano_node_metrics_forks_int We see a slowly but steadily growing number of forks. A natural and expected thing in Ouroboros proability base decentralised slot leader election. But we also see the growth rate increasing in the reward-calculation window, which means this is caused by extended propagation delays and certain blocks produced without having received the very last block in time.

enter image description here

Update 2022-01-11

Since pools started upgrading to node version 1.33, the previously described 48h window for the reward calculation becomes re-scheduled to cover almost the whole epoch. This will result in less impact on the 48h window.

The following graphs shows this together with a second effect we currently see reducing the CDF1 ratio: more large blocks.

At the moment of this writing, ~ 64% of all nodes was updated to v 1.33

enter image description here

6
  • Note: Starting with node version 1.33.x, the behavior in the 48-hour window will improve significantly because the reward calculation is implemented differently. Jan 3 at 12:01
  • A huge "thank you" for the detailed explanation. 8 hours after restarting the bp node with version 1.33.0, the numbers aren't the greatest cardano_node_metrics_blockfetchclient_blockdelay_cdfThree 0.9925925925925926 cardano_node_metrics_blockfetchclient_blockdelay_cdfOne 0.275 cardano_node_metrics_blockfetchclient_lateblocks 4 cardano_node_metrics_blockfetchclient_blockdelay_cdfFive 0.9972222222222222 cardano_node_metrics_blockfetchclient_blockdelay_s 1.75081688 Jan 10 at 2:28
  • Fantastic answer.
    – 74d4
    Jan 10 at 14:25
  • @OmahaCardanoPool I just updated the answer, to explain the newly introduced behaviour and effect of node version 1.33 At the timestamp of your comment, my nodes had CDF1 ratios between 0.3 and 0.6 so your 0.275 is not a real bad value for that specific moment. Jan 11 at 19:20
  • @Markus-CLIO1 Since the Feb 7 restart of the BP node, there have been 101 late blocks. Reason to be concerned? cardano_node_metrics_blockfetchclient_lateblocks 101 Feb 20 at 23:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.