Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Upgrade localstack version and remove pinned dependencies (#107)
# Upgrade localstack version and remove pinned dependencies In commit bdfb0d6 we pinned AWS dependencies, since we weren't sure if `eventq` was ready for upgrading them. Also they dropped Ruby 2.5 support. Since we have moved to have no pinned dependencies if possible, each environment can choose the best and most up to date dependencies available, so we can keep the Ruby 2.5 support and also ensure using the latest AWS libraries in Ruby 3. Also we removed the fixed localstack version, as in version 1.4.0 they addressed the multiregion support. # Refactor AWS integration specs Upgrading our localstack version showed how brittle our integration tests are, because newer localstack versions react a bit slower (read: 100+ ms to pick up a message instead of 10+ ms). Since we used `sleep` statements a ton to wait "just the right amount of time" to exceed the TTL, specs became flaky or consistently failed because the worker was picking them up later than intended. Looking deeper into it showed that our approach was flawed from the get go. While we name the variable in specs `retry_attempt_count` it is actually a `call_count`. Therefore all `sleeps` were off by one iteration. In the past this created the following patterns in specs: * 0.0s Boot worker * 0.1s `sleep 1` to let worker pick up message * 0.2s Worker processes message * 1.1s Check message has been processed * 1.1s `sleep 1` to wait for retry * 1.2s Worker processes retry Sometimes there was just 10 ms between assertion and worker processing the message. Since the worker is on a thread, this could have failed with older localstack versions as well. With the newer and slower localstack version most specs looked like this: * 0.0s Boot worker * 0.1s `sleep 1` to let worker pick up message * 0.8s Worker processes message * 1.1s Check message has been processed * 1.1s `sleep 1` to wait for retry * 2.0s Worker processes retry * 2.1s `sleep 1` to wait for 2nd retry * 3.1s Check for 2nd retry fails as worker has not picked up 2nd retry yet The flaw here is that `visibility_timeout` does not tell when the message is picked up again, but for how long it is NOT PICKED UP again. So while we need to _at least_ wait for the `visibility_timeout` in specs, we cannot know when the worker actually comes around to pick the message up again. If we were using bigger numbers, like minutes for the `visibility_timeout` rather than seconds, the 0.1 - 1.9 seconds sometimes needed to pick up a message would not matter, but also would slow down specs to a halt. The second flaw is that we are basically testing libraries here. We should trust AWS to ensure messages are not picked up for the `visibility_timeout` we set. Instead we should only test if the right `visibility_timeout` numbers are calculated for the libraries to use. As a result we have adjusted the specs to use a `visibility_timeout` of zero whenever possible to speed up specs. However, we note down the calculated number to ensure the right one is normally passed on to AWS. This allows us to use the exponential backoff strategy, without waiting expontentially long. In order to remove the need for `sleep` in specs, as the operation to test is run asynchroneously, we introduced a thread based queue to wait for results on. This way, as soon as a result is pushed to the queue in another thread, the thread waiting for results immediately picks it up and continues running the spec and assertions. By adding a decorator around a method that is invoked after each time a message was processed, we can simply say `wait_for_message_processed` to ensure the spec wait just long enough to continue. To test `visibility_timeout` we can now use `aws_visibility_timeout_queue`, which gets pushed the `visibility_timeout` calculated for each call, e.g. ``` expect(aws_visibility_timeout_queue.pop).to eq(call: 5, visibility_timeout: 5) ``` Using these techniques we sped up specs considerably, made them much more reliable and still kept the integration part as real AWS libraries are used. Also we can still integrate with a real AWS account instead of using `localstack`. # Set log level to FATAL during spec runs Since the logger is printing messages to STDOUT, all DEBUG, INFO and ERROR logs produced during tests are included in the test output. Since we rely heavily on failing to process a message on purpose, we have littered the output with ERROR logs and traces as well. By setting the log level to FATAL during spec runs, we ensure that we only see the spec output during test runs. --------- Co-authored-by: Florian Pilz <[email protected]>
- Loading branch information
1 parent
f47b8c7
commit 37bfbe3