compute,storage: expose wallclock lag history in SQL

[teskje]

This PR adds a new builtin source, mz_wallclock_lag_history, that reports the minute-by-minute maximum wallclock lag (diff between write frontier and wallclock time) observed for each storage and compute collection. Both controllers take part in populating the history, each contributing lags for the collections they are responsible for.

mz_wallclock_lag_history reports per-replica lag for compute objects and global lag (based on the persisted frontier) for storage collections. Where there is overlap (i.e. for materialized views), both are reported. There is also a builtin view, mz_wallclock_global_lag_history that exposes only global lags for all collections (by taking the minimum of all recorded lags). This structure was previously discussed here.

To give the console fast access to the wallclock lag data, an index view mz_wallclock_global_lag_recent_history is also provided. This view has a temporal filter to 1 day, to keep the index's memory usage in check.

The first commit is from #29423 and can be ignored here.

Motivation

• This PR adds a known-desirable feature.

Part of MaterializeInc/database-issues#8235

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[shepherdlybot]

Mitigations

Completing required mitigations increases Resilience Coverage.

• (Required) Code Review 🔍 Detected
• (Required) Feature Flag
• (Required) Integration Test 🔍 Detected
• (Required) Observability
• (Required) QA Review 🔍 Detected
• (Required) Run Nightly Tests
• Unit Test

Risk Summary:

The pull request carries a high risk score of 83, driven by predictors such as the average line count in files and executable lines within files, with two file hotspots involved. Historically, PRs with these predictors have been 152% more likely to cause a bug compared to the repository baseline. Notably, the observed bug trend in the repository is currently decreasing.

Note: The risk score is not based on semantic analysis but on historical predictors of bug occurrence in the repository. The attributes above were deemed the strongest predictors based on that history. Predictors and the score may change as the PR evolves in code, time, and review activity.

Bug Hotspots:
What's This?

File	Percentile
../src/lib.rs	98
../src/controller.rs	96

[benesch]

:chefskiss:

[benesch]

This looks awesome, @teskje — thank you! I didn't review the storage controller changes in full detail, so ideally @petrosagg has time to take a second look there tomorrow, but if not I'm comfortable with you merging anyway so that this makes the release cut.

[teskje]

I deployed this branch on my staging env and let it run for a day to observe the memory usage of the new index.

This is without any user collections, so just the system sources/tables/indexes/subscribes. There are 303 of these currently (plus a bunch of slow-path SELECTs randomly showing up).

The arrangement size stabilizes at 26 MiB on my env. That's roughly 88 KiB per collection. The environment with the most collections right now has 1258, which would end up at ~108 MiB.

If we assume twice that amount, to account for the higher per-record usage for the switch from uint8 to Interval, and other overheads our arrangement size metrics don't track, we end up at 216 MiB in the worst case (and 65 MiB for the median). All of the production environments easily have that much to spare, so it's safe to merge this PR.

Even if somehow my estimations are wildly off, we still have two means to adjust this in prod: scaling up the size of mz_catalog_server, and reducing the interval with which new wallclock lag events are emitted.

[petrosagg]

LGTM!

[teskje]

TFTRs!