SHiP: fails to recover from a corrupted last entry in the log file

[linh2931]

This is related to test failures 3, 4, and 5 in tests/ship_restart_test.py, #983. Once the issue is resolved, re-enable the tests.

If the last entry of SHiP log is corrupted, SHiP plugin tries to repair the log by discarding the last entry and updating _end_block (becoming smaller). Startup continues. But later, SHiP node detects blocks being skipped due to end_block not in consistent with the head block in the state:

error 2024-10-30T00:59:30.707 nodeos    state_history_plugin.c:150    on_accepted_block    ] fc::exception: 3110000 plugin_exception:   Plugin exception
block 154 skips over block 153 in /home/lh/work/ship/build/TestLogs/ship_restart_test116718/node_01/state-history/chain_state_history.  log
    {"b":154,"e":153,"name":"/home/lh/work/ship/build/TestLogs/ship_restart_test116718/node_01/state-history/chain_state_history.log"}
    nodeos  log.hpp:220 pack_and_write_entry

info  2024-10-30T00:59:30.707 nodeos    main.cpp:172                  operator()           ] appbase quit called
warn  2024-10-30T00:59:30.707 nodeos    controller.hpp:523            emit                 ] /home/lh/work/ship/libraries/chain/        controller.cpp:3425 controller_emit_signal_exception: 3140002 state_history_write_exception: State history write error
State history encountered an Error which it cannot recover from.  Please resolve the error and relaunch the process
    {}
    nodeos  state_history_plugin.cpp:155 on_accepted_block

error 2024-10-30T00:59:30.707 nodeos    controller.cpp:3835           apply_block          ] e.to_detail_string(): 3140002              state_history_write_exception: State history write error

[bhazzard]

This might should be part of a broader effort to improve crash recovery of nodeos

[bhazzard]

@bhazzard Create new issues to add truncation of SHiP log and trace api and close this as wont-fix.