Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(batch-exports): Improve batch export alerting by checking for missing runs #27013

Merged
merged 5 commits into from
Dec 19, 2024

Conversation

rossgray
Copy link
Contributor

Problem

We've had an issue recently where Temporal hasn't scheduled a workflow run for some reason and we didn't detect this.

Changes

Update existing monitoring job to also check for missing batch export runs.

We will log a warning message when this happens, which we can then create an alert for in alertmanager.

Does this work well for both Cloud and self-hosted?

it doesn't have an impact

How did you test this code?

Updated existing tests + added a new one

@rossgray rossgray requested a review from timgl December 18, 2024 13:58
@rossgray rossgray force-pushed the ross/batch-exports-alerts-missing-runs branch from 677b9c5 to e24b9b1 Compare December 18, 2024 15:54
@rossgray rossgray merged commit 4c0162e into master Dec 19, 2024
92 checks passed
@rossgray rossgray deleted the ross/batch-exports-alerts-missing-runs branch December 19, 2024 09:48
Copy link

sentry-io bot commented Dec 20, 2024

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ OperationalError: consuming input failed: query_wait_timeout posthog.batch_exports.service in <listcomp> View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants