-
-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Avoid random failures in system tests #5319
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jorg-vr
changed the title
Avoid race conditions in tests
Avoid system tests impacting eachother
Jan 22, 2024
pdawyndt
changed the title
Avoid system tests impacting eachother
Avoid system tests impacting each other
Jan 22, 2024
This reverts commit cdc755a.
jorg-vr
changed the title
Avoid system tests impacting each other
Avoid random failures in system tests
Jan 29, 2024
This reverts commit 6609925.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request is an effort to avoid occasional failures in the system tests. The global solution was increasing the maximal wait time for finding elements on a page to 30 seconds.
This does not slow down succeeding test. It will slowdown failing tests.
There were multiple points of failure encountered while working on this pr:
This finally led me to conclude that the issues were not test specific.
This pr also has multiple minor changes to help avoid these random failures:
annotation-text
count to change before checking backend change. This should avoid a timing issues if the frontend changes has not yet propagated to the backend.The tests now use a json exercise with the normal feedback table renderer. The assumption here was that the code tab rendering might have infrequent errors for the code tab.The code tab is clicked reference by id instead of by text. The idea was that a text search for the button may sometimes fails due to the number tag also in the buttonThe maximum waiting time for finding the code tab has been increased to 5 seconds (maybe the page is slow to load)I have added a sanity check to first find the body element and the page wrapper before searching for the code tab. I added this to debug, not as a fix, but it is the only thing that worked as a fix.Increase wait time for scratchpad test. This test deserves some additional worst case wait time for the setup of webworker.I have tested this pr by rerunning the tests 10 times after each change.
Notes
I found it very odd, that although it varies in which test a failure happens, there is always only a single failing test.
Increasing the number of tests, by duplicating one of the failing tests 40 times, did not seem to increase the frequency of failures. (It took me 10 tries to reproduce a failure with the 40 added tests)
When I started working on this issue I mostly encountered the 'code tab not found' failure.
When that error stopped occurring I started seeing the 'question count value incorrect' failure.
Now that that that seems fixed, the #4562 failure reappeared after not having seen it for weeks.
Increasing the allowed waiting time for that test, let another totally unrelated test fail...
Except for the incorrect count the issue is always the same. An item that is present on the page is not found.
Always only a single test failing