Pectra Devnet 5 Hive / Execution Spec Tests

[siladu]

https://hive.pectra-devnet-5.ethpandaops.io/

Tests: https://github.com/ethereum/execution-spec-tests/releases/tag/pectra-devnet-5%40v1.1.0

[siladu]

#8080 adds adds the prereleased spec tests 5efd396
to the draft devnet-5 branch...

[siladu]

Latest results for Prague fork using https://github.com/ethereum/execution-spec-tests/tree/pectra-devnet-5%40v1.3.0

Running with EEST

uv run fill --fork=Prague --evm-bin=../besu/build/install/besu/bin/evmtool --evm-dump-dir ./prague

10 failed, 11062 passed, 18 skipped, 106 deselected, 16158 warnings, 65 errors in 4532.28s (1:15:32)

FAILED tests/cancun/eip4844_blobs/test_blob_txs.py::test_invalid_tx_max_fee_per_blob_gas_state[fork_Prague-insufficient_max_fee_per_blob_gas-state_test] - ethereum_test_specs.helpers.TransactionUnexpectedSuccessError:
FAILED tests/prague/eip2935_historical_block_hashes_from_state/test_block_hashes.py::test_block_hashes_history[fork_Prague-blockchain_test_engine-full_history_plus_one_check_blockhash_first] - requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=62579): Read timed out. (read timeout=5)
FAILED tests/prague/eip2935_historical_block_hashes_from_state/test_contract_deployment.py::test_system_contract_deployment[fork_CancunToPragueAtTime15k-blockchain_test-deploy_after_fork] - ethereum_test_types.types.Alloc.MissingAccountError: Account missing from allocation 0x0f792be4b0c0cb4dae440ef133e90c0ecd48cccc
FAILED tests/prague/eip2935_historical_block_hashes_from_state/test_contract_deployment.py::test_system_contract_deployment[fork_CancunToPragueAtTime15k-blockchain_test_engine-deploy_after_fork] - ethereum_test_types.types.Alloc.MissingAccountError: Account missing from allocation 0x0f792be4b0c0cb4dae440ef133e90c0ecd48cccc
FAILED tests/shanghai/eip3860_initcode/test_initcode.py::test_contract_creating_tx[fork_Prague-blockchain_test-over_limit_zeros] - ethereum_test_specs.helpers.TransactionExceptionMismatchError:
FAILED tests/shanghai/eip3860_initcode/test_initcode.py::test_contract_creating_tx[fork_Prague-blockchain_test-over_limit_ones] - ethereum_test_specs.helpers.TransactionExceptionMismatchError:
FAILED tests/shanghai/eip3860_initcode/test_initcode.py::test_contract_creating_tx[fork_Prague-blockchain_test_engine-over_limit_zeros] - ethereum_test_specs.helpers.TransactionExceptionMismatchError:
FAILED tests/shanghai/eip3860_initcode/test_initcode.py::test_contract_creating_tx[fork_Prague-blockchain_test_engine-over_limit_ones] - ethereum_test_specs.helpers.TransactionExceptionMismatchError:
FAILED tests/shanghai/eip3860_initcode/test_initcode.py::test_contract_creating_tx[fork_Prague-state_test-over_limit_zeros] - ethereum_test_specs.helpers.TransactionExceptionMismatchError:
FAILED tests/shanghai/eip3860_initcode/test_initcode.py::test_contract_creating_tx[fork_Prague-state_test-over_limit_ones] - ethereum_test_specs.helpers.TransactionExceptionMismatchError:

Analysis so far

• tests/shanghai/eip3860_initcode fixed by Use correct exception message for besu ethereum/execution-spec-tests#1121
• tests/cancun/eip4844_blobs fixed by Fix Prague execution-spec-tests #8167

---

Running with gradle reference tests

referenceTestImplementation 'ethereum:execution-spec-tests:[email protected]:[email protected]'

./gradlew ethereum:referenceTests:referenceTests
...

66212 tests completed, 393 failed, 6258 skipped

[siladu]

With the two PRs linked above applied, I get 3 errors left, all eip2935. These tests don't fail in the referenceTests runner so I suspect it's just a test infra issue.

Full rerun:

uv run fill --fork=Prague --evm-bin=../besu/build/install/besu/bin/evmtool --evm-dump-dir ./prague

FAILED tests/frontier/opcodes/test_push.py::test_stack_overflow[fork_Prague-state_test-stack_height_1025-PUSH32] - requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('POST / HTTP/1.1\r\n'))
FAILED tests/prague/eip2935_historical_block_hashes_from_state/test_block_hashes.py::test_block_hashes_history[fork_Prague-blockchain_test_engine-full_history_plus_one_check_blockhash_first] - requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=63983): Read timed out. (read timeout=5)
FAILED tests/prague/eip2935_historical_block_hashes_from_state/test_contract_deployment.py::test_system_contract_deployment[fork_CancunToPragueAtTime15k-blockchain_test-deploy_after_fork] - ethereum_test_types.types.Alloc.MissingAccountError: Account missing from allocation 0x0f792be4b0c0cb4dae440ef133e90c0ecd48cccc
FAILED tests/prague/eip2935_historical_block_hashes_from_state/test_contract_deployment.py::test_system_contract_deployment[fork_CancunToPragueAtTime15k-blockchain_test_engine-deploy_after_fork] - ethereum_test_types.types.Alloc.MissingAccountError: Account missing from allocation 0x0f792be4b0c0cb4dae440ef133e90c0ecd48cccc
============================================================= 4 failed, 11068 passed, 18 skipped, 106 deselected, 16158 warnings in 5921.96s (1:38:41)

but I think the first one was a transient error, reran and it passed:

uv run fill --fork=Prague -k test_push --evm-bin=../besu/build/install/besu/bin/evmtool --evm-dump-dir ./prague tests/frontier

288 passed, 63 deselected, 480 warnings in 40.10s