Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Non-deterministic test failures #415

Open
sbellem opened this issue Dec 13, 2019 · 1 comment
Open

Non-deterministic test failures #415

sbellem opened this issue Dec 13, 2019 · 1 comment
Assignees
@sbellem

Comments

@sbellem
Copy link
Collaborator

sbellem commented Dec 13, 2019 •
edited
Loading

Some tests appear to fail in non-deterministic ways, meaning that for the exact same code version/hash a test sometimes passes and sometimes fails.

I'll soon provide examples, and will look into how we may possibly address the problem.

Resources

The trio library may be worth looking at, at least conceptually, for how it handles such non-deterministic failures. From their contributing section docs:

Tests MUST pass deterministically. Flakey tests make for miserable developers. One common source of indeterminism is scheduler ordering; if you’re having trouble with this, then trio.testing provides powerful tools to help control ordering, like trio.testing.wait_all_tasks_blocked(), trio.testing.Sequencer, and trio.testing.MockClock (usually used as a fixture: async def test_whatever(autojump_clock): ...). And if you need more tools than this then we should add them.

In addition to the interesting test tools to help prevent non-deterministic tests failures, it is in interesting to see how they handle these test failures when they do occur, on Travis CI for instance:

Another interesting issue is on finding and reproducing heisenbugs in concurrent programs, it has many links to interesting resource:
@sbellem sbellem self-assigned this Dec 13, 2019
@sbellem
Copy link
Collaborator Author

sbellem commented Dec 14, 2019 •
edited
Loading

Example:

docker run -it $tag \
    pytest --verbose \
           --numprocesses=auto \
           --cov \
           --cov-report=term-missing \
           --cov-report=xml \
           -Wignore::DeprecationWarning

/opt/venv/lib/python3.7/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment.

  warner(PytestBenchmarkWarning(text))

============================= test session starts ==============================

platform linux -- Python 3.7.5, pytest-5.3.1, py-1.8.0, pluggy-0.13.1 -- /opt/venv/bin/python
cachedir: .pytest_cache
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/usr/src/HoneyBadgerMPC/.hypothesis/examples')
rootdir: /usr/src/HoneyBadgerMPC, inifile: pytest.ini
plugins: mock-1.13.0, web3-5.4.0, asyncio-0.10.0, xdist-1.30.0, benchmark-3.2.2, cov-2.8.1, forked-1.1.3, env-0.6.2, hypothesis-4.50.6
[gw0] linux Python 3.7.5 cwd: /usr/src/HoneyBadgerMPC
[gw1] linux Python 3.7.5 cwd: /usr/src/HoneyBadgerMPC
[gw0] Python 3.7.5 (default, Nov 23 2019, 06:10:46)  -- [GCC 8.3.0]
[gw1] Python 3.7.5 (default, Nov 23 2019, 06:10:46)  -- [GCC 8.3.0]
gw0 [260] / gw1 [260]
scheduling tests via LoadScheduling
tests/test_asynchromix.py::test_phase1 
tests/test_asynchromix.py::test_butterfly_network Fatal Python error: Illegal instruction
Thread 0x00007f0e63472700 (most recent call first):
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 400 in read
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 432 in from_io
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 220 in run
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn

Current thread 0x00007f0e6395f740 (most recent call first):

  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 224 in _write_polys
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 357 in generate_values
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 606 in _generate
  File "/usr/src/HoneyBadgerMPC/honeybadgermpc/preprocessing.py", line 633 in generate_powers
  File "/usr/src/HoneyBadgerMPC/tests/test_asynchromix.py", line 48 in test_phase1
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88 in _run
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 1763 in _run_once
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 534 in run_forever
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 566 in run_until_complete
  File "/opt/venv/lib/python3.7/site-packages/pytest_asyncio/plugin.py", line 158 in pytest_pyfunc_call
  File "/opt/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/opt/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/opt/venv/lib/python3.7/site-packages/_pytest/python.py", line 1435 in runtest
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 131 in pytest_runtest_call
  File "/opt/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/opt/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 207 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 234 in from_call
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 207 in call_runtest_hook
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 182 in call_and_report
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 96 in runtestprotocol
  File "/opt/venv/lib/python3.7/site-packages/_pytest/runner.py", line 81 in pytest_runtest_protocol
  File "/opt/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/opt/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/opt/venv/lib/python3.7/site-packages/xdist/remote.py", line 87 in run_one_test
  File "/opt/venv/lib/python3.7/site-packages/xdist/remote.py", line 70 in pytest_runtestloop
  File "/opt/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/opt/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/opt/venv/lib/python3.7/site-packages/_pytest/main.py", line 246 in _main
  File "/opt/venv/lib/python3.7/site-packages/_pytest/main.py", line 196 in wrap_session
  File "/opt/venv/lib/python3.7/site-packages/_pytest/main.py", line 239 in pytest_cmdline_main
  File "/opt/venv/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/opt/venv/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/opt/venv/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/opt/venv/lib/python3.7/site-packages/xdist/remote.py", line 258 in <module>
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 1084 in executetask
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 220 in run
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 267 in integrate_as_primary_thread
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 1060 in serve
  File "/opt/venv/lib/python3.7/site-packages/execnet/gateway_base.py", line 1554 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>
[gw1] node down: Not properly terminated
[gw1] [  0%] FAILED tests/test_asynchromix.py::test_phase1 

replacing crashed worker gw1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sbellem sbellem

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sbellem