test: Fix tests to reduce chance of 'database is locked' error. #4651

[giacob500]

This pull request addresses and improves the stability of the tests related to database access, specifically targeting the database is locked error in SQLite. The error typically occurs when database operations are attempted simultaneously, leading to contention and test failures.

Key Changes:

• Retry Logic for SQLite Queries: Added a helper function execute_with_retry to gracefully handle SQLite database lock errors. This function retries the query a few times with a short delay before failing, reducing the chance of encountering lock issues.
• Improved Test Handling: In tests that interact with the database (e.g., test_import_export_json), calls to the database now utilize the retry logic to ensure smooth execution even in the event of temporary locking.
• Synchronous Grouping: Updated test marking for better clarity and organization. Tests that require external systems or involve long operations have been moved into the synchronous group to ensure that they run without asynchronous-related issues.

These improvements should help reduce flakiness in the test suite, particularly for long-running tests interacting with the SQLite database.

Related Issue:

• Fixes issue test: Fix tests to reduce chance of "database is locked" error. #4651

[giacob500]

I decided to create my own marker @pytest.mark.synchronous to make the role of synchronous functions explicit, however pytest should not require it. In theory, by removing line 30, 40 of test/test_cvedb.py and the updates made to setup.cfg we should be able to obtain the same result.

Let me know your considerations, as I don't want to add bloat code. Thanks.

[terriko]

I've set the tests to run, but... I think if we're going to do a retry, we should be doing it in cve-bin-tool itself, not only the test suite. Could you look into moving your code there?

[giacob500]

Hi Terri, when you say

Could you look into moving your code there?

do you mean removing the retry logic from the tests entirely and implementing it solely in the core cve-bin-tool code? Or should it remain in both the tests and the core? Just want to ensure I follow your intended approach. Thanks!

[terriko]

do you mean removing the retry logic from the tests entirely and implementing it solely in the core cve-bin-tool code? Or should it remain in both the tests and the core? Just want to ensure I follow your intended approach. Thanks!

The former: there should be no retry at all in the tests because the retry will be handled in the code (probably in cvedb.py or util.py). The idea is that the whole tool should get better at not getting stuck if the database is locked. I don't think it happens as often during normal operation since you aren't running as many database-changing operations at once, but sometimes people execute multiple scans at once and forget to use -u never on them so I imagine database lock errors happen sometimes outside of the test suite.

That suggests another usability improvement: if we do exceed the retries and finally time out, we should have the final error specifically suggest to people that they use -u never on parallel jobs so they know to fix it in future runs.

Thanks for clarifying!