Adjust queues to prioritize account syncs, handle missing current day values #1682
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There have been several complaints in #1524 of how certain account graphs do not include the latest day's balance. While I have not been able to reproduce this, I think a potential cause is self-hosted queues skipping the background jobs that generate the balances.
Insufficient DB connections
Currently, we are running self-hosted GoodJob jobs in
async
mode, which means the queue is running in the same process as the web server. This is sufficient for most self hosted instances since the number of requests to the server are very low (~1-5 total users).That said, when in
async
mode, the required DB pool size increases:This means we need
11
total DB connections to support a self hosted instance. My guess is that the missing / sporadic graph balances is a result of jobs exhausting the connection pool, timing out, and not syncing the latest balances.This PR increases the default DB pool size to
11
to accommodate this.If running GoodJob in
external
mode (default for production, and used for our hosted instance), each process requires a different number of DB connections:Loading data state
To further protect against the case where an account is missing / not properly syncing the latest balances, I have added a "loading" state to the graphs to let the user know that we're still calculating the latest balances.
In an ideal state, this shouldn't be shown often because we have logic to sync user accounts immediately when they login. The thinking here is that we'd rather show "loading" than an invalid balance graph that leads a user to think their finances are something different than they actually are.
Queue latencies
In addition to the above changes, I have implemented 3 queues based on latency:
latency_low
- jobs that take ~30 seconds or lesslatency_medium
- jobs that take 1-2 minutes (i.e. Sync accounts job)latency_high
- jobs that take 5+ minutes (i.e. EnrichDataJob)This should drastically speed up account syncs by offloading the expensive and slow EnrichDataJob to a separate worker pool entirely.