Skip to content

20241022

20241022: Fix fee catchup after restart (#2160)
@jbearer jbearer tagged this 22 Oct 17:46 
* Add regression test for fee catchup after restart

This test reproduces an issue discovered in staging, in which, after
a staggered restart of all nodes, the network has forgotten the fee
state for all accounts except those which actively produced blocks
during the restart. Since the state is needed for undecided blocks,
it cannot directly be obtained from persisted merklized state, which
is only available for decided blocks.

* Add bulk endpoint for account catchup

* Enable account catchup from persisted decided storage

* Cache fetched state in validated state map after fetching from database

* Support migration from existing nodes populating leaf_hash column

* Update HotShot  to enable caching of fetched account states

* Rename migration after merge

* Add proposal fetching task

* Return leaf when fetching account state so we can always add it to the HotShot state map

* Fix deadlock in restart tests

During experimentation I changed the restart tests from starting up
all nodes in parallel to one at a time. This broke the no-CDN tests
because, in some cases, you need multiple nodes to restart before
libp2p can become ready, but this blocks the initialization of the
first restarted node.

Switched it back to starting up in parallel and things are working
again.

* Update sequencer/api/catchup.toml

Co-authored-by: Mathis <[email protected]>

* Avoid dropping TaskList, to avoid blocking executor thread in drop handler

* Exit proposal fetching task if it becomes obsolete

The simplest, most robust change was to make the task a flat retry
loop, instead of having a nested retry loop for the fetch itself,
and to update the anchor view each iteration of the loop, so that
the loop exits if we reach a decide and the anchor view becomes
more recent than the view we are fetching.

* Enable frontier catchup from storage (#2183)

* Enable frontier catchup from storage

* Prevent recursive state reconstruction

* Prefetch accounts used for paying fees during state reconstruction

* Remove no-longer-necessary catchup impl for DB transactions

* Completely disable catchup when not required (NullStateCatchup)

* Fix backoff CLI parsing

* Use the proper chain config during state reconstruction

Closes #2186

* Add comment

* Fix migration conflict

* Make migrations consistent with release-gambit

---------

Co-authored-by: Mathis <[email protected]>
Assets 2
Loading