Test Failure: restart-scenarios-test-hard_replay

[heifner]

https://github.com/AntelopeIO/spring/actions/runs/9100452880/job/25015943882

  File "/__w/spring/spring/build/tests/restart-scenarios-test.py", line 52, in <module>
    if cluster.launch(pnodes=pnodes, totalNodes=total_nodes, topo=topo, delay=delay, activateIF=activateIF) is False:
  File "/__w/spring/spring/build/tests/TestHarness/Cluster.py", line 532, in launch
    if not self.bootstrap(launcher, self.biosNode, self.startedNodesCount, prodCount + sharedProducers, totalProducers, pfSetupPolicy, onlyBios, onlySetProds, loadSystemContract, activateIF, biosFinalizer, signatureProviderForNonProducer):
  File "/__w/spring/spring/build/tests/TestHarness/Cluster.py", line 1194, in bootstrap
    ret=Utils.waitForBool(lam)
  File "/__w/spring/spring/build/tests/TestHarness/testUtils.py", line 271, in waitForBool
    ret=Utils.waitForObj(myLam, timeout, sleepTime, reporter=reporter)
  File "/__w/spring/spring/build/tests/TestHarness/testUtils.py", line 261, in waitForObj
    raise RuntimeError('waitForObj reached 60 second timeout')
RuntimeError: waitForObj reached 60 second timeout

A quick look it appears that node_00 was not able to establish a connection to peers at startup. By the time it was able to connect and sync, its last synced block had latency: 9183 ms which is too much for it to start producing. Since it doesn't produce the test times out waiting. node_00 has producers: defproducera - defproducerf.

[arhag]

Might be related to #399?

[bhazzard]

Haven't seen this fail recently.