Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logging and metrics for slow HTTP requests #1540

Merged
merged 2 commits into from
Jun 7, 2024

Conversation

jbearer
Copy link
Member

@jbearer jbearer commented Jun 3, 2024

In Cappuccino, we observed poor performance while DA nodes were syncing, due to a missing index on the payload hash field causing Postgres queries to be slow. I had previously seen occasional CPU spikes and slow requests from running the nasty client, so I suspect this issue would have been caught earlier, before deploying to production, if these types of performance issues uncovered by the nasty client were easier to diagnose (turns out we weren't doing queries by payload hash at all). This PR aims to make it so by improving visibility into slow requests (even those that don't actually time out) via logging and metrics.

This PR:

  • Adds queries by payload hash
  • Adds a new nasty client parameter to warn, but not error, if requests are too slow. This can be set fairly aggressively to catch even subtle performance errors, like 1s.
  • Adds three new metrics:
    • Slow request threshold, which helps to interpret the slow request counter
    • Counter of slow requests
    • Histogram of request latencies

This PR does not:

Fix any performance issues

Key places to review:

nasty_client.rs

How to test this PR:

Run just demo-native. Navigate to http://localhost:24011/status/metrics. Observe the http_slow_requests counter and the http_request_latency histogram.

Base automatically changed from jb/nasty-client to main June 7, 2024 14:30
@jbearer jbearer force-pushed the jb/nasty-client-slow-requests branch from b9fc39c to be24ee7 Compare June 7, 2024 14:34
@jbearer jbearer marked this pull request as ready for review June 7, 2024 14:34
@jbearer jbearer enabled auto-merge June 7, 2024 14:34
@jbearer jbearer merged commit 5f2c5bf into main Jun 7, 2024
13 checks passed
@jbearer jbearer deleted the jb/nasty-client-slow-requests branch June 7, 2024 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants