1

We've started using the 1.35.x node docker images in preparation for Vasil. I always start with the testnet before deploying to production. In our testnet deployment, the new images will not work. While synching with the chain, they hang up. If we restart them, we get a communication error. You can see these below:

Step 1: run node docker image from scratch (all relevant volumes are first deleted)

b8e82085:cardano.node.ChainDB:Notice:35] [2022-08-14 11:41:24.19 UTC] Chain extended, new tip: c7380283205467161f2a587da610ea79b482347ecc9a459f9ecf3030c03e5a6a at slot 63417270
[b8e82085:cardano.node.ChainDB:Error:35] [2022-08-14 11:41:24.48 UTC] Invalid block 708745ea94515b679223a5514e6e6dba068208e5b79780134206f74e1e79e727 at slot 63419637: ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (MalformedScriptWitnesses (fromList [ScriptHash "eee7bfb25cfbbbcd31a1d5d648429ff919430ee5d06d8d056e48d605"])))))])})))))))
[b8e82085:cardano.node.ChainDB:Info:35] [2022-08-14 11:41:24.48 UTC] Valid candidate ce9c9431882310df559099e2bfa3e6eb31c95606d83ffc05568df7f863d4db60 at slot 63419598

We let the system run for 12 hours after that last line. No further activity occured. The node was hung up.

Step 2: restart the docker image

Since the node was hung up, we killed the container and started it up again. We left the volume mounts intact. The node then threw the following after boilerplate initalization:

[973dad53:cardano.node.DnsSubscription:Error:233] [2022-08-14 16:57:26.08 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Application Exception: 3.131.32.242:3001 InvalidBlock (At (Block {blockPointSlot = SlotNo 63419662, blockPointHash = 38dc2f5dbb184b504eda086a5f64c19931a985bf2246b74ed18e796712cff67a})) 708745ea94515b679223a5514e6e6dba068208e5b79780134206f74e1e79e727 (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (MalformedScriptWitnesses (fromList [ScriptHash "eee7bfb25cfbbbcd31a1d5d648429ff919430ee5d06d8d056e48d605"])))))])})))))))))
[973dad53:cardano.node.ErrorPolicy:Warning:119] [2022-08-14 16:57:26.08 UTC] IP 3.131.32.242:3001 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (InvalidBlock (At (Block {blockPointSlot = SlotNo 63419662, blockPointHash = 38dc2f5dbb184b504eda086a5f64c19931a985bf2246b74ed18e796712cff67a})) 708745ea94515b679223a5514e6e6dba068208e5b79780134206f74e1e79e727 (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (MalformedScriptWitnesses (fromList [ScriptHash "eee7bfb25cfbbbcd31a1d5d648429ff919430ee5d06d8d056e48d605"])))))])})))))))))))) 200s 200s
[973dad53:cardano.node.ChainDB:Info:102] [2022-08-14 16:57:34.32 UTC] Took ledger snapshot DiskSnapshot {dsNumber = 63343083, dsSuffix = Nothing} at 1f46697e362676f795cfe4608217c805bde9830fea2c304381cb6c9853b1a074 at slot 63343083
[973dad53:cardano.node.DnsSubscription:Warning:121] [2022-08-14 16:57:35.78 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Unsupported remote target address [2a05:d014:e00:a202:0:1:0:2]:3001
[973dad53:cardano.node.DnsSubscription:Notice:265] [2022-08-14 16:57:35.78 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Connection Attempt Start, destination 3.131.32.242:3001
[973dad53:cardano.node.DnsSubscription:Warning:121] [2022-08-14 16:57:35.80 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Unsupported remote target address [2a05:d014:e00:a202:0:1:0:3]:3001
[973dad53:cardano.node.DnsSubscription:Notice:266] [2022-08-14 16:57:35.80 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Connection Attempt Start, destination 3.133.220.20:3001
[973dad53:cardano.node.DnsSubscription:Warning:121] [2022-08-14 16:57:35.83 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Unsupported remote target address [2a05:d014:e00:a201:0:1:0:2]:3001
[973dad53:cardano.node.DnsSubscription:Notice:267] [2022-08-14 16:57:35.83 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Connection Attempt Start, destination 54.199.108.71:3001
[973dad53:cardano.node.DnsSubscription:Notice:265] [2022-08-14 16:57:35.83 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Connection Attempt End, destination 3.131.32.242:3001 outcome: ConnectSuccessLast
[973dad53:cardano.node.ErrorPolicy:Notice:119] [2022-08-14 16:57:35.83 UTC] IP 3.133.220.20:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s
[973dad53:cardano.node.ErrorPolicy:Notice:119] [2022-08-14 16:57:35.83 UTC] IP 54.199.108.71:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s

It seems to be complaining about connection with Cardano relays. This doesn't make sense to us since we if we repeat the above process it will start synchronizing again but eventually fail in the same way which suggests communication with the relays is possible. This machine has a stable internet connection. In fact it's running on the same machine as our production system which is working fine (still on an older version of the docker image).

We have tried 1.35.1 and 1.35.3. They both do the same thing. These nodes are connected to the testnet.

Does anyone know why we can't get this node running? We're out of ideas for what could be wrong.

Improve this question

3 Answers 3

Reset to default
2

We have tried 1.35.1 and 1.35.3.

You should ONLY use node 1.35.2 on testnet. 1.35.1 produced unforeseen invalid block on testnet network, which was fixed on node 1.35.2 and rules updated to discard the invalid blocks (to prevent fork) at node 1.35.3. Thus, the testnet network is in a state where you cannot use 1.35.3 or above, and 1.35.1 would produce invalid blocks.

The only node version that will be usable on that network is 1.35.2. IO have announced they will be moving to new networks (that have already been spun) soon, you can follow those via environments page. As regards docker, I dont use it myself but if these configs are not readily available, you can mount them on.

Improve this answer
1
  • 1
    Would have been nice for the developers who published this image to throw a simple error if you pass the testnet parameter in.
    – melchoir55
    Aug 16, 2022 at 16:06
0

Wanted to reply, and state I have this same error building the 1.35.3 node and cli from source on github I spent a few hours trying to sort this out as my issues started while I was on 1.35.2. After hours of updating my system, and rebuilding the node, and re-syncing DB from scratch; I found this article, appears 1.35.3 won't work for now on testnet??? https://forum.cardano.org/t/upgrading-to-1-35-3-all-3-nodes-down/105926/15 I have another testnet node running 1.35.3 and is up to date????

My actual block producer node is back to 1.35.2 and producing blocks again...

Improve this answer
0

Please note that the docker-compose.yml file on 13.0.4 release of cardano-db-sync pulls the 1.35.3 version of node (https://github.com/input-output-hk/cardano-db-sync/blob/13.0.4/docker-compose.yml) This should be updated to use 1.35.2 if running testnet.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.