Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make warmup to respect dual read mode, and separate warmup configs for indis #962

Merged
merged 16 commits into from
Jan 5, 2024

Conversation

bohhyang
Copy link
Contributor

@bohhyang bohhyang commented Dec 22, 2023

Summary

Current warmup starts for both ZK and xDS regardless of dual read mode, and the logic of fetching dual read mode (service data) blocks the warmup timeout which fails app startup.

This change:

  1. make warmup to respect the dual read mode per d2 service.
  2. move the blocking service data fetching logic into a separate task and set a timeout, so that it won't blocks the warmup timeout callback to be called (so startup still has a chance to complete).
  3. add separate configs for Indis warmup timeout and concurrent requests. We could tune them in the future as needed depending on how rate limiting on Indis observer will go. And wait for indis warmup (thus use the indis warmup timeout config) under observer-only mode during startup (instead of waiting for zk warmup).
  4. add checks to dual read load balancer monitoring to skip putting duplicate data into the report caches (which incorrectly increase outOfSync metric).

Testing Done

  1. added unit tests
  2. tested with d2-proxy warming up some 200 services under different dual read modes, see the corresponding container PR for logs.

@bohhyang bohhyang requested a review from shivamgupta1 January 2, 2024 16:46
@bohhyang bohhyang changed the title Make warm-up to respect dual read mode, and separate warmup configs for indis Make warmup to respect dual read mode, and separate warmup configs for indis Jan 2, 2024
@bohhyang bohhyang requested a review from shivamgupta1 January 3, 2024 23:28
@bohhyang bohhyang requested a review from shivamgupta1 January 4, 2024 16:25
@bohhyang bohhyang force-pushed the boyang/indisWarmup branch from bcdb328 to 929d4ab Compare January 4, 2024 18:07
@bohhyang bohhyang merged commit 146e1f2 into master Jan 5, 2024
2 checks passed
@bohhyang bohhyang deleted the boyang/indisWarmup branch January 5, 2024 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants