4

Have recently rotated KES following steps. I cannot get any block to get adopted despite being selected as a slot leader number of times already. As you can see Guild LiveView tool shows that KES kes are valid. But the logs from cardano-node show otherwise.

enter image description here

Jun 17 16:39:29 wladyslawa bash[1021152]:

[wladysla:cardano.node.ChainDB:Error:82] [2021-06-17 15:39:29.01 UTC] Invalid block 95def156e23112ebe3f9c6832ac96059d10ff5c446bdbbbc275b2698dfc336d0 at slot 32378078: ExtValidationErrorHeader (HeaderProtocolError (HardForkValidationErrFromEra S (S (S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (OcertFailure (InvalidKesSignatureOCERT 249 243 6 "Reject"))]}))))))

More errors:

maciej@ola:~/Downloads$ grep -ir ":error" logs.txt  -C 1
Jun 17 14:00:38 wladyslawa bash[1021152]: [wladysla:cardano.node.Forge:Info:92] [2021-06-17 13:00:38.00 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 3.2368547e7)])),("credentials",String "Cardano")]
Jun 17 14:00:38 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Error:73120] [2021-06-17 13:00:38.48 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Application Exception: 192.168.0.12:6000 ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (ServerAgency TokNext TokMustReply)
Jun 17 14:00:38 wladyslawa bash[1021152]: [wladysla:cardano.node.ErrorPolicy:Notice:100] [2021-06-17 13:00:38.48 UTC] IP 192.168.0.12:6000 ErrorPolicySuspendConsumer (Just (ApplicationExceptionTrace ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (ServerAgency TokNext TokMustReply))) 20s
--
Jun 17 14:04:25 wladyslawa bash[1021152]: [wladysla:cardano.node.Forge:Info:92] [2021-06-17 13:04:25.00 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 3.2368774e7)])),("credentials",String "Cardano")]
Jun 17 14:04:25 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Error:99511] [2021-06-17 13:04:25.20 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Application Exception: 192.168.0.12:6000 ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (ServerAgency TokNext TokMustReply)
Jun 17 14:04:25 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Info:99511] [2021-06-17 13:04:25.20 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Closed socket to 192.168.0.12:6000
--
Jun 17 14:05:56 wladyslawa bash[1021152]: [wladysla:cardano.node.Forge:Info:92] [2021-06-17 13:05:56.00 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 3.2368865e7)])),("credentials",String "Cardano")]
Jun 17 14:05:56 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Error:99547] [2021-06-17 13:05:56.21 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Application Exception: 192.168.0.12:6000 ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (ServerAgency TokNext TokMustReply)
Jun 17 14:05:56 wladyslawa bash[1021152]: [wladysla:cardano.node.ErrorPolicy:Notice:100] [2021-06-17 13:05:56.21 UTC] IP 192.168.0.12:6000 ErrorPolicySuspendConsumer (Just (ApplicationExceptionTrace ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (ServerAgency TokNext TokMustReply))) 20s
--
Jun 17 14:05:57 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Info:102] [2021-06-17 13:05:57.21 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Skipping peer 192.168.0.12:6000
Jun 17 14:05:57 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Error:102] [2021-06-17 13:05:57.21 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Failed to start all required subscriptions
Jun 17 14:05:57 wladyslawa bash[1021152]: [wladysla:cardano.node.Mempool:Info:99577] [2021-06-17 13:05:57.23 UTC] fromList [("tx",Object (fromList [("txid",String "txid: TxId {_unTxId = SafeHash \"fb5f593592735aa5507b2881705e902611729620968b4c890718cb775ce9d4fc\"}")])),("mempoolSize",Object (fromList [("bytes",Number 87490.0),("numTxs",Number 152.0)])),("kind",String "TraceMempoolRejectedTx"),("err",Object (fromList [("badInputs",Array [String "d22db824b458406b322b8e2620768bd3c5ff1120208de2fbce57d5b553a69ff5#1"]),("consumed",Object (fromList [("lovelace",Number 0.0),("policies",Object (fromList []))])),("error",String "This transaction consumed Value 0 (fromList []) but produced Value 284069674 (fromList [(PolicyID {policyID = ScriptHash \"defca4b95934aec82b818cbafd4bd0ab45511c1d690b58759f07d9b1\"},fromList [(\"VyFi\",24900)])])"),("kind",String "ValueNotConservedUTxO"),("produced",Object (fromList [("lovelace",Number 2.84069674e8),("policies",Object (fromList [("defca4b95934aec82b818cbafd4bd0ab45511c1d690b58759f07d9b1",Object (fromList [("56794669",Number 24900.0)]))]))]))]))]
--
Jun 17 14:06:07 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Info:102] [2021-06-17 13:06:07.21 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Skipping peer 192.168.0.12:6000
Jun 17 14:06:07 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Error:102] [2021-06-17 13:06:07.21 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Failed to start all required subscriptions
Jun 17 14:06:08 wladyslawa bash[1021152]: [wladysla:cardano.node.LeadershipCheck:Info:92] [2021-06-17 13:06:08.00 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":5.040153e-2,"slot":32368877,"delegMapSize":644495,"utxoSize":2660038,"credentials":"Cardano"}
--
Jun 17 16:39:29 wladyslawa bash[1021152]: [wladysla:cardano.node.Forge:Info:92] [2021-06-17 15:39:29.00 UTC] fromList [("val",Object (fromList [("kind",String "TraceForgedBlock"),("slot",Number 3.2378078e7)])),("credentials",String "Cardano")]
Jun 17 16:39:29 wladyslawa bash[1021152]: [wladysla:cardano.node.ChainDB:Error:82] [2021-06-17 15:39:29.01 UTC] Invalid block 95def156e23112ebe3f9c6832ac96059d10ff5c446bdbbbc275b2698dfc336d0 at slot 32378078: ExtValidationErrorHeader (HeaderProtocolError (HardForkValidationErrFromEra S (S (S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (OcertFailure (InvalidKesSignatureOCERT 249 243 6 "Reject"))]}))))))
Jun 17 16:39:29 wladyslawa bash[1021152]: [wladysla:cardano.node.ChainDB:Info:82] [2021-06-17 15:39:29.01 UTC] Valid candidate 82548006ab042d802e1f885e028ea93038282d028b9530999e9dfc63473ff247 at slot 32378076
Jun 17 16:39:29 wladyslawa bash[1021152]: [wladysla:cardano.node.Forge:Error:92] [2021-06-17 15:39:29.01 UTC] fromList [("val",Object (fromList [("kind",String "TraceForgedInvalidBlock"),("reason",Object (fromList [("error",Object (fromList [("error",Object (fromList [("failures",Array [Object (fromList [("error",String "Reject"),("kind",String "InvalidKesSignatureOCERT"),("opCertExpectedKESEvolutions",String "6"),("opCertKESCurrentPeriod",String "249"),("opCertKESStartPeriod",String "243")])]),("kind",String "ChainTransitionError")])),("kind",String "HeaderProtocolError")])),("kind",String "ValidationError")])),("slot",Number 3.2378078e7)])),("credentials",String "Cardano")]
Jun 17 16:39:30 wladyslawa bash[1021152]: [wladysla:cardano.node.LeadershipCheck:Info:92] [2021-06-17 15:39:30.00 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":5.0332207e-2,"slot":32378079,"delegMapSize":644766,"utxoSize":2661388,"credentials":"Cardano"}

All logs at Dropbox.

Update:

I followed the directions to make sure that the opCertExpectedKESEvolutions matched the actual evolutions. The command cardano-cli text-view decode-cbor --in-file <path>/op.cert | grep int | head -1 returns 6 but now the expected evolutions is 8???

That did not work. I am getting the same error again.

Jul 01 16:02:50 wladyslawa bash[680275]: [wladysla:cardano.node.Forge:Error:115] [2021-07-01 15:02:50.01 UTC] fromList [("val",Object (fromList [("kind",String "TraceForgedInvalidBlock"),("reason",Object (fromList [("error",Object (fromList [("error",Object (fromList [("failures",Array [Object (fromList [("error",String "Reject"),("kind",String "InvalidKesSignatureOCERT"),("opCertExpectedKESEvolutions",String "8"),("opCertKESCurrentPeriod",String "259"),("opCertKESStartPeriod",String "251")])]),("kind",String "ChainTransitionError")])),("kind",String "HeaderProtocolError")])),("kind",String "ValidationError")])),("slot",Number 3.3585479e7)])),("credentials",String "Cardano")]
5

gLiveView can only tell you what your current KES key's start/expiry time is. But another base property of KES key is the number of times your key was rotated. You can check your current key using cardano-cli text-view decode-cbor --in-file <path>/op.cert | grep int | head -1. Your current cert can only be higher, (not lower) than the one used for your previous blocks. The upcoming public pgrest API on guild-operators website (currently in alpha stage) will allow you to query your last created block's OCert number for validation too.

PS: You can rotate multiple times to a higher counter (so for example, you can go rotate until counter is 10) just fine, as it will still be higher than your previous block's OCert counter.

5
  • Thank you. That number shows 2 which is what I would expect given that I rotated KES twice already. Would you mind clarify further because I cannot see how I can use this number for further debugging.
    – matcheek
    Jun 18 at 9:08
  • As per your original thread, it lookslike the counter expected is 6 , not 2. So there is block on chain where your pool's opcert counter was 5 already, and you need to rotate it 4 more times for your index to be higher.
    – RdLrT
    Jun 18 at 11:09
  • I think you can also just open the counter file and manually increase it to something higher. Jun 18 at 20:43
  • My pooled was elected a block leader yesterday. Unfortunately that changes that you proposed and I enacted did not work. Same error, though expected KES evolutions is now 8.
    – matcheek
    Jul 2 at 8:58
  • Please specify both your current counter, new counter in opcert and whether your new KES is generated from same cold key
    – RdLrT
    Jul 3 at 11:09
1

We are seeing similar error. We only have rotated the cert once and that was after minting our first block. @RdLrT this command shows just 1.

cardano-cli.sh "text-view decode-cbor --in-file node.cert"  | grep int | head -1
      01  # int(1)

But the error says

"opCertExpectedKESEvolutions",String "17" Does this mean I have to rotate it 16 more times?

cardano-node    | [93f57906:cardano.node.ChainDB:Info:537] [2021-06-17 11:48:08.00 UTC] Valid candidate 1ef53e05e0c8ac9a35519dedc61e974036cb66fbf15a6ecc4c3420fa0db9f8ef at slot 32364102
cardano-node    | [93f57906:cardano.node.Forge:Error:547] [2021-06-17 11:48:08.01 UTC] fromList [("val",Object (fromList [("kind",String "TraceForgedInvalidBlock"),("reason",Object (fromList [("error",Object (fromList [("error",Object (fromList [("failures",Array [Object (fromList [("error",String "Reject"),("kind",String "InvalidKesSignatureOCERT"),("opCertExpectedKESEvolutions",String "17"),("opCertKESCurrentPeriod",String "249"),("opCertKESStartPeriod",String "232")])]),("kind",String "ChainTransitionError")])),("kind",String "HeaderProtocolError")])),("kind",String "ValidationError")])),("slot",Number 3.2364197e7)])),("credentials",String "Cardano")]
cardano-node    | [93f57906:cardano.node.LeadershipCheck:Info:547] [2021-06-17 11:48:09.00 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.991215e-2,"slot":32364198,"delegMapSize":644407,"utxoSize":2659418,"credentials":"Cardano"}
4
  • I followed that advice but there is no way to verify if it solves the problem before any blocks are actually minted. I should know the answer within next week or two.
    – matcheek
    Jun 23 at 17:55
  • 1
    This appears to be a new question, not really an answer to the original question. Please consider whether you should make it a separate question, update or delete it. Welcome aboard, I hope my feedback is helpful.
    – gRebel
    Jun 23 at 21:28
  • @gRebel the problem is exactly the same and didn't want to open a new thread on this to avoid missed context. Jun 24 at 1:13
  • @user2107938 Based on the log message, yes (or you can verify in your dbsync instance too)
    – RdLrT
    Jun 24 at 6:18
0

If it's not the counter issue mentioned above, I had a similar error in the past:

[cardano-:cardano.node.Forge:Error:195] [2021-03-15 23:36:47.00 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceForgedInvalidBlock"),("reason",Object (fromList [("kind",String "ValidationError"),("error",Object (fromList [("kind",String "HeaderProtocolError"),("error",Object (fromList [("kind",String "ChainTransitionError"),("failures",Array [Object (fromList [("kind",String "InvalidKesSignatureOCERT"),("error",String "Reject"),("opCertKESCurrentPeriod",String "165"),("opCertExpectedKESEvolutions",String "1"),("opCertKESStartPeriod",String "164")])])]))]))])),("slot",Number 2.1482191e7)]))]

I believe the cause was that I only copied the updated kes file to the producer, and not the new node.cert file. I re-copied both files over to the producer and restarted, and the blocks started working correctly.

4
  • Thank you. May I ask you. The steps that I followed only mention about copying node.cert to the prodcution environment. coincashew.com/coins/overview-ada/… What steps did you follow? Did you manage to find any officals Cardano docs for this?
    – matcheek
    Jun 19 at 19:03
  • @matcheek - The issue is already proven to be as per first reply based on log extract and your first reply about current ocert number. It would best for you to rotate your KES 4 more times (so that current is 6 or above) and verify next block assignment is created correctly.
    – RdLrT
    Jun 20 at 0:46
  • @RdLrT Done. Do you know how to verify if KES is correct now before waiting for another minted block?
    – matcheek
    Jun 20 at 8:03
  • I've no idea why it only mentions one, I'm pretty certain you need to copy both! I followed the instructions on the official Cardano site (which has just been updated and is hard to find things now), and I just repeated the bit I assumed was required for KES rotation. I don't know of any way to verify things are correct before minting a block though (I hit this issue on testnet where I had a large delegation and was able to verify fairly quickly). Jun 20 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.