Test suite clean up #3385

JDBetteridge · 2024-02-02T16:52:43Z

Description

Switches to having MPI executed 'on the outside'. This means we are no longer bound to MPICH (the source of huge build times and MacOS bugs) and also makes things much faster.
Parallel tests do not oversubscribe the runners.
Various associated bug fixes since parallel tests are not run in isolation.

TODOs

Fix remaining failing tests
Add make check and clean up Makefile (and use in CI + mention on install page)
Remove --download-mpich from everywhere, means that MPI is a system dependency

connorjward · 2024-02-02T17:38:33Z

~~This is cool, but isn't it a bad idea to effectively remove test coverage? If CI doesn't run all the tests no one will.~~

~~I can see this being useful in the context of a bigger change where we run the test suite with a number of Firedrake configurations and only one of them would run these slow tests.~~

tests/regression/test_ensembleparallelism.py

.github/workflows/build.yml

github-actions · 2024-09-12T13:53:27Z

	Tests	Passed ✅	Skipped ⏭️	Failed ❌
Firedrake complex	31 ran	27 passed	4 skipped	0 failed

github-actions · 2024-09-12T13:59:51Z

	Tests	Passed ✅	Skipped ⏭️	Failed ❌
Firedrake real	29 ran	27 passed	2 skipped	0 failed

connorjward

Generally very happy with this.

.github/workflows/build.yml

.test_durations

firedrake/parameters.py

firedrake/slate/slac/compiler.py

firedrake/tsfc_interface.py

tests/demos/test_demos_run.py

tests/output/test_io_mesh.py

tests/slate/test_hdg_poisson.py

connorjward

Leaving notes for someone (most likely me) to refer to in future. In summary:

Need to rebase/merge in master.
Tweaks to Makefile and build.yml.

connorjward · 2024-10-31T14:14:07Z

.github/workflows/build.yml

+            -o faulthandler_timeout=1860 \
+            --junit-xml=firedrake2_\$MPISPAWN_TASK_ID1.xml \
+            -m "parallel[\$MPISPAWN_WORLD_SIZE] and not broken" \
+            -v tests


TODO: "dogfood" (bleh) Makefile and use a matrix to massively cut down on boilerplate

connorjward · 2024-10-31T14:17:08Z

Makefile

+.PHONY: test_smoke
+test_smoke:
+	@echo "    Running the bare minimum smoke tests"
+	@python -m pytest -k "poisson_strong or stokes_mini or dg_advection" -v tests/regression/


It would be better to use MPI on the "outside" here for the parallel tests so this can be run to check things on HPC

connorjward · 2024-10-31T14:17:43Z

Makefile

-endif
+# Requires pytest and pytest-mpi only
+.PHONY: test_serial
+test_serial:


Terrible name! This runs all the parallel tests too!

connorjward · 2024-10-31T14:18:39Z

Makefile

+
+# Requires pytest and pytest-mpi only
+.PHONY: test_smoke
+test_smoke:


bikeshedding: I prefer make smoke_tests or make smoketests

connorjward · 2024-10-31T14:19:28Z

Makefile

+	done
+
+.PHONY: _test_large_world_test
+_test_large_world_tests:


I'm not sure why we have small_world and large_world tests separately.

firedrake/slate/slac/kernel_builder.py

…_tests

tests/firedrake/conftest.py

tests/firedrake/multigrid/test_netgen_gmg.py

tests/firedrake/vertexonly/test_poisson_inverse_conductivity.py

tests/firedrake/equation_bcs/test_equation_bcs.py

tests/firedrake/demos/test_notebooks_run.py

tests/firedrake/conftest.py

JDBetteridge · 2025-01-09T17:40:22Z

Why would you remove the slow markers? They are really useful if you want to run the test suite faster (especially in parallel!) and are also there to highlight tests that can be sped up.

connorjward · 2025-01-09T17:49:04Z

Why would you remove the slow markers?

Sorry to undo your hard work but I don't see the benefit in having them:

They are really useful if you want to run the test suite faster (especially in parallel!)

But it doesn't run all of the tests. It is useful to be able to run only a few fast tests (a la make check) but I can't imagine a situation where I would want to run most-but-not-all of the tests.

and are also there to highlight tests that can be sped up.

We already know which tests are slow from .test_durations and from the

============================ slowest 200 durations =============================

output from pytest. This is another source of truth and I think it will immediately bit-rot.

connorjward · 2025-01-10T15:56:03Z

tests/firedrake/output/test_adjoint_disk_checkpointing.py

@@ -61,17 +60,15 @@ def adjoint_example(mesh):


 @pytest.mark.skipcomplex
-# Waiting on stable parallel decompositions through disk checkpointing.
-@pytest.mark.xfail


TODO: What is going on here

…er_tests

I'm in charge now.

JDBetteridge added the DO NOT MERGE label Feb 2, 2024

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from ecb9628 to f3685b7 Compare February 9, 2024 19:21

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from f3685b7 to f848c6f Compare March 5, 2024 13:40

JDBetteridge force-pushed the JDBetteridge/faster_tests branch 2 times, most recently from 93a0aae to a20fc9d Compare June 7, 2024 13:52

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from a5f614f to 27b9af3 Compare July 19, 2024 16:25

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from 2a7f468 to 6ed1774 Compare August 18, 2024 13:44

JDBetteridge force-pushed the JDBetteridge/faster_tests branch 2 times, most recently from 8132b64 to 1122a3b Compare August 31, 2024 18:37

JDBetteridge added enhancement performance and removed DO NOT MERGE labels Sep 4, 2024

JDBetteridge self-assigned this Sep 4, 2024

JDBetteridge added the bug label Sep 4, 2024

JHopeCollins previously requested changes Sep 10, 2024

View reviewed changes

tests/regression/test_ensembleparallelism.py Outdated Show resolved Hide resolved

JDBetteridge commented Sep 11, 2024

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

JDBetteridge force-pushed the JDBetteridge/faster_tests branch 2 times, most recently from 9d5f056 to df4aea3 Compare September 12, 2024 13:04

JDBetteridge marked this pull request as ready for review September 24, 2024 14:23

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from bf317f5 to ef87021 Compare October 2, 2024 21:57

connorjward changed the title ~~Mark and skip slow tests~~ Test suite clean up Oct 3, 2024

connorjward reviewed Oct 3, 2024

View reviewed changes

JDBetteridge force-pushed the JDBetteridge/faster_tests branch from 8435a69 to 10f7f0c Compare October 8, 2024 14:02

JDBetteridge and others added 4 commits October 10, 2024 15:38

Draft changes to pyop2.caching

fcdeace

Change package branch

056ba30

Just notes

314ff8e

WIP

3f272d6

Stop using pytest-mpi branch

130b7b8

connorjward previously requested changes Oct 31, 2024

View reviewed changes

JHopeCollins mentioned this pull request Nov 18, 2024

BUG: Mesh partitioners are not deterministic across different Ensemble members #3866

Open

Merge remote-tracking branch 'origin/master' into JDBetteridge/faster…

ea03f73

…_tests

connorjward assigned connorjward and unassigned JDBetteridge Jan 9, 2025

connorjward reviewed Jan 9, 2025

View reviewed changes

tweaks to build.yml, need to check before running

788c910

connorjward added the macOS label Jan 9, 2025

connorjward added 2 commits January 10, 2025 15:54

Remove slow markers

d62b3a9

More to remove

2f18719

connorjward reviewed Jan 10, 2025

View reviewed changes

connorjward added 3 commits January 10, 2025 16:07

Split apart build.yml

399694f

Merge branch 'connorjward/from-testing-branch' into JDBetteridge/fast…

897b9bb

…er_tests

cleanup

ac1a2cf

connorjward added 11 commits January 10, 2025 17:55

Experimental: add testing action

d29c285

fixup, maybe works

9f34981

testing

1ff0f91

fixup

7c2afde

fixup?

ebda6c0

fixups, I think

1eab6a8

fail slow and remove flaky test

611c92b

linting

31356d4

Stop using a custom action

6969c39

for debugging, much smaller timeout

77fa692

silly fixup

6587b16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test suite clean up #3385

Test suite clean up #3385

JDBetteridge commented Feb 2, 2024 •

edited by connorjward

Loading

connorjward commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

connorjward left a comment

connorjward left a comment

connorjward Oct 31, 2024

connorjward Oct 31, 2024

connorjward Oct 31, 2024

connorjward Oct 31, 2024

connorjward Oct 31, 2024

JDBetteridge commented Jan 9, 2025

connorjward commented Jan 9, 2025

connorjward Jan 10, 2025

Test suite clean up #3385

Are you sure you want to change the base?

Test suite clean up #3385

Conversation

JDBetteridge commented Feb 2, 2024 • edited by connorjward Loading

Description

TODOs

connorjward commented Feb 2, 2024 • edited Loading

github-actions bot commented Sep 12, 2024 • edited Loading

github-actions bot commented Sep 12, 2024 • edited Loading

connorjward left a comment

Choose a reason for hiding this comment

connorjward left a comment

Choose a reason for hiding this comment

connorjward Oct 31, 2024

Choose a reason for hiding this comment

connorjward Oct 31, 2024

Choose a reason for hiding this comment

connorjward Oct 31, 2024

Choose a reason for hiding this comment

connorjward Oct 31, 2024

Choose a reason for hiding this comment

connorjward Oct 31, 2024

Choose a reason for hiding this comment

JDBetteridge commented Jan 9, 2025

connorjward commented Jan 9, 2025

connorjward Jan 10, 2025

Choose a reason for hiding this comment

JDBetteridge commented Feb 2, 2024 •

edited by connorjward

Loading

connorjward commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading