cluster-controller: reconfiguration observability + SHOW CLUSTERS by aljoscha · Pull Request #37628 · MaterializeInc/materialize

aljoscha · 2026-07-14T09:33:29Z

NOTE: This is PR4 in a stack of PRs, the full branch is at #36738.

Surface cluster reconfigurations and autoscaling state in SQL so a background ALTER CLUSTER is observable. All dark: the durable reconfiguration and burst records only ever move under the enable_cluster_controller master gate (default off), so an ordinary cluster has nothing in flight and the new relations are empty for it.

Two introspection relations

Both are builtin materialized views in mz_catalog_server, computed from the raw catalog (mz_catalog_raw) so they are pure functions of durable state, and both are indexed on cluster_id. Each carries its structural payload as JSON so the schema is stable as strategies grow.

mz_internal.mz_cluster_reconfigurations — one row per cluster that carries a graceful reconfiguration record: cluster_id, a status that is in-progress while the controller converges and then a terminal finalized / timed-out / cancelled / resource-exhausted, the deadline, the on_timeout action, the target config shape as JSON, and changes: a JSON object diffing target against the realized config per dimension (size, replication_factor, availability_zones, logging), holding the target value for each dimension that differs. Both sides of the diff come from the same raw catalog document, so the jsonb comparison is canonical and matches the routing's shape-equality. The record is retained after it settles, so the latest outcome stays inspectable until a later reconfiguration overwrites it (changes then reads as "how target differs from realized now": empty after cut-over, the abandoned diff after a rollback). The realized (current) shape stays in mz_clusters. Status values are kebab-case, matching the audit log's transition values and the catalog's other multi-word values (materialized-view, on-refresh). The CASE mappings pass unmapped variants through verbatim rather than falling to NULL, where the ASSERT NOT NULL would error every read of the relation (and of SHOW CLUSTERS, which joins it).
mz_internal.mz_cluster_auto_scaling_strategies — one row per cluster that has an AUTO SCALING STRATEGY configured or an autoscaling action running: the configured strategy as JSON and the in-flight state (the active burst, when one is up) as JSON or NULL. Nothing can configure a strategy until the burst PR lands its SQL surface, so its user-facing docs ship with that PR and this PR marks the relation RELATION_SPEC_UNDOCUMENTED.

SHOW CLUSTERS

SHOW CLUSTERS gains one nullable activity column (between replicas and comment): a human-readable one-line summary of any in-flight reconfiguration or burst, NULL when the cluster is steady. A single SHOW CLUSTERS answers "is something in progress" without the user knowing about the two relations. The reconfiguration summary is rendered from changes, naming exactly the changed dimensions (with values where they read well), so a reconfiguration that does not touch the size never reads as a size change:

reconfiguring size to scale=2,workers=8, replication factor to 2, introspection settings

Settled reconfiguration records are retained, so the summary matches only status = 'in-progress'; the auto-scaling relation's state is set only while a burst runs. Neither needs mz_now(), so mz_show_clusters stays indexable.

Plumbing

The two new builtin indexes change the SQL fingerprint of mz_indexes (it inlines the builtin-index set as VALUES), which requires an explicit builtin-schema-migration replacement step. NOTE for the final rebase: the step version must track the workspace's current dev version until this ships in a release, upstream version bumps silently re-break upgrade tests otherwise.
The new builtin MVs cannot advance their write frontier in read-only (0dt) mode, so the caught-up check's exclusion walk in coord.rs now also seeds from new builtin MVs, excluding their transitive dependents (here: the indexed mz_show_clusters).

Tests

An observability section in cluster-controller.td asserts the two relations, the activity column, and the reconfiguration audit-log trail end to end (finalized, cancel-race, rollback-at-deadline, forced COMMIT cut-over, and a deterministically frozen in-progress record that alters size, replication factor, and introspection interval in one statement to pin the multi-dimension rendering). Gate-off column assertions in show_clusters.slt. Catalog-snapshot slt/testdrive expectations and the mz_internal system-catalog docs updated.

Motivation

Implements the two-relation observability surface and the SHOW CLUSTERS activity column from doc/developer/design/20260522_cluster_autoscaling.md.

User-visible change: SHOW CLUSTERS output gains an activity column, and mz_internal gains the mz_cluster_reconfigurations relation (plus the yet-undocumented mz_cluster_auto_scaling_strategies).

aljoscha · 2026-07-14T09:40:30Z


-                // Migrated MVs can't make progress in read-only mode. Exclude them and all their
-                // transitive dependents.
+                // A collection that can't advance its write frontier in read-only mode also


we need to boil down this large wall of text to be way more concise please

Boiled down to 7 lines (from 18), keeping the two collection kinds and the accepted-blip tradeoff.

aljoscha · 2026-07-14T09:41:50Z

+                "The config shape the cluster is reconfiguring to, as JSON: `size`, `replication_factor`, `availability_zones`, and `logging`. The realized (current) shape is in `mz_clusters`.",
+            ),
+        ]),
+        // One row per managed cluster that carries a graceful reconfiguration


I think we can also boil down these comments on the new builtins, big time please

Boiled both down: the reconfigurations block is now 8 lines (retention, JSON-null presence check, kebab values, and the ELSE-passthrough rationale since that one is load-bearing), the auto-scaling one is 5.

aljoscha · 2026-07-14T09:42:17Z

        .with_column("comment", SqlScalarType::String.nullable(false))
        .finish(),
    column_comments: BTreeMap::new(),
+    // `activity` is a one-line, human-readable summary built from the two base


also boil down this one please

Done, 5 lines: the in-progress-only match, no-mz_now indexability, and the NULL-concat gotcha.

aljoscha · 2026-07-14T09:53:28Z

+> SELECT recon.status FROM mz_internal.mz_cluster_reconfigurations recon JOIN mz_clusters ON mz_clusters.id = recon.cluster_id WHERE mz_clusters.name = 'cc_rollback'
+finalized
+
+# An in-flight reconfiguration observed deterministically: freeze the controller by


didn't we move away from this pattern in earlier changes of our STACK, we can't be 100% sure that nothing interferes here, or at least we have to change this to use mz_wait as some other tests already do, no?

Yes, good catch, that was a leftover racy pattern: cut-over could in principle land before the gate flip. Reworked to the established mz_sleep pattern from the deadline scenarios above: a sleeping MV in the cluster keeps the target set from hydrating, so the record deterministically stays in-progress (deadline at 600s, out of reach), and dropping the view lets it hydrate and settle. No gate flipping anymore.

Correction to my last reply: the unblock-and-converge step did not survive contact with CI. The sleep runs on the replica's worker thread, so dropping the view cannot unwedge the replicas, and the cut-over never comes (Testdrive 9 caught it). The scenario now ends by dropping the cluster with the record in flight instead, which must succeed and leaves the audit lifecycle at started. The deterministic in-progress observation is unchanged.

mtabebe

Clean change, and scope to just the introspection. One question around how we could have null records (the comment wasn't particularly clear).

mtabebe · 2026-07-14T23:18:47Z

            MZ_CATALOG_SCHEMA,
            "mz_audit_events",
        ),
+        // Required because we added the `mz_cluster_reconfigurations_ind` and


mtabebe · 2026-07-14T23:22:38Z

+        FROM mz_internal.mz_catalog_raw
+        WHERE
+            data->>'kind' = 'Cluster' AND
+            data->'value'->'config'->'variant'->'Managed' IS NOT NULL


When are the cases when it is null? The comment above wasn't very informative

Two separate nulls get filtered here. The Managed IS NOT NULL in the CTE drops unmanaged clusters: their config lives under the Unmanaged variant, so the ->'Managed' lookup returns SQL NULL, and only managed clusters can carry a reconfiguration record. The != 'null' in the outer WHERE handles managed clusters that have never gracefully reconfigured: the durable field is optional, and mz_catalog_raw serializes the unset field as an explicit JSON null rather than omitting the key, so an IS NOT NULL check would not filter it. I sharpened the comment above the view to say exactly this.

⚙️ — aj the shiny one

mtabebe · 2026-07-14T23:24:35Z

-                // transitive dependents.
+                    new_builtin_collections.iter().copied().collect();
+
+                // A collection that can't advance its write frontier in read-only mode


Thanks for improving the comments

**NOTE: This is PR4 in a stack of PRs, the full branch is at MaterializeInc#36738 Surface cluster reconfigurations and autoscaling state in SQL so a background ALTER CLUSTER is observable. All dark: the durable reconfiguration and burst records only ever move under the `enable_cluster_controller` master gate, so an ordinary cluster has nothing in flight and the new relations are empty for it. Two introspection relations. Both are builtin materialized views in `mz_catalog_server`, computed from the raw catalog (`mz_catalog_raw`) so they are pure functions of durable state, and both are indexed on `cluster_id`. Each carries its structural payload as JSON so the schema is stable as strategies grow. * `mz_internal.mz_cluster_reconfigurations`, one row per cluster that carries a graceful reconfiguration record: `cluster_id`, a `status` that is `in-progress` while the controller converges and then a terminal `finalized` / `timed-out` / `cancelled` / `resource-exhausted`, the `deadline`, the `on_timeout` action, and the `target` config shape as JSON. The record is retained after it settles, so the latest outcome stays inspectable until a later reconfiguration overwrites it. The realized (current) shape stays in `mz_clusters`. Status values are kebab-case, matching the audit log's transition values and the catalog's other multi-word values, and the CASE mappings pass unmapped variants through verbatim rather than falling to NULL, where the ASSERT NOT NULL would error every read of the relation. * `mz_internal.mz_cluster_auto_scaling_strategies`, one row per cluster that has an AUTO SCALING STRATEGY configured or an autoscaling action running: `cluster_id`, the configured `strategy` as JSON, and the in-flight `state` (the active burst, when one is up) as JSON or NULL. Nothing can configure a strategy until the burst PR lands its SQL surface, so its user-facing docs ship with that PR and this PR marks the relation RELATION_SPEC_UNDOCUMENTED. SHOW CLUSTERS. `mz_show_clusters` LEFT JOINs both relations to add one `activity` column, a human-readable one-line summary of any in-flight reconfiguration or burst, so a single SHOW CLUSTERS answers "is something in progress" without the user knowing about the two relations. Settled reconfiguration records are retained, so the join matches only `status = 'in-progress'`; the auto-scaling relation's `state` is set only while a burst runs. Neither needs `mz_now()`, so the view stays indexable. The two new builtin indexes change the SQL fingerprint of `mz_indexes` (it inlines the builtin-index set as VALUES), which requires an explicit builtin-schema-migration replacement step. The step version must track the workspace's current dev version until this ships in a release. The new builtin MVs also need the read-only caught-up check to account for their dependents, so `coord.rs` seeds new-builtin-MV dependents into the exclusion walk. Tests: an observability section in `cluster-controller.td` asserting the two relations, the SHOW CLUSTERS `activity` column, and the reconfiguration audit-log trail end to end, gate-off column assertions in `show_clusters.slt`, and the catalog-snapshot slt/testdrive expectations plus the `mz_internal` system-catalog doc updated for the new relation, its index, and the SHOW CLUSTERS `activity` column. Implements the two-relation observability surface and the SHOW CLUSTERS activity column from `doc/developer/design/20260522_cluster_autoscaling.md`. Co-Authored-By: Claude <noreply@anthropic.com>

A graceful reconfiguration can change more than size: availability zones and introspection settings are shape dimensions of their own, and a replication factor change rides along when combined with a shape change in one statement. SHOW CLUSTERS rendered activity as "reconfiguring to <size>" regardless, which reads as a no-op, or worse a lie, for reconfigurations that do not touch the size. mz_cluster_reconfigurations gains a `changes` column: a JSON object holding the target value for each dimension in which the target differs from the realized config. Both sides of the diff come from the same raw catalog document, so the jsonb comparison is canonical and matches the routing shape-equality (an AZ reorder counts as a change in both). On retained records the column reads as "how the target differs from the realized config now": empty once the target applied, the abandoned diff after a rollback. SHOW CLUSTERS builds activity from `changes`, naming only the changed dimensions, with values where they read well: reconfiguring size to scale=2,workers=8, replication factor to 2, availability zones, introspection settings The frozen-reconfiguration testdrive scenario now alters size, replication factor, and introspection interval in one statement to pin the multi-dimension rendering. Co-Authored-By: Claude <noreply@anthropic.com>

aljoscha requested review from a team as code owners July 14, 2026 09:33

aljoscha commented Jul 14, 2026

View reviewed changes

aljoscha force-pushed the adapter-cluster-controller-observability branch 2 times, most recently from b41a71a to 5e1d9b7 Compare July 14, 2026 10:29

aljoscha added the ci-nightly PR CI control: also trigger Nightly label Jul 14, 2026

mtabebe self-requested a review July 14, 2026 15:52

mtabebe approved these changes Jul 14, 2026

View reviewed changes

aljoscha force-pushed the adapter-cluster-controller-observability branch from 5e1d9b7 to 648f663 Compare July 15, 2026 09:11

aljoscha force-pushed the adapter-cluster-controller-observability branch from 648f663 to 373f778 Compare July 15, 2026 09:29

aljoscha force-pushed the adapter-cluster-controller-observability branch from 373f778 to a2e2859 Compare July 15, 2026 09:33

aljoscha merged commit 6c07c97 into MaterializeInc:main Jul 15, 2026
355 of 372 checks passed

Uh oh!

Conversation

aljoscha commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Two introspection relations

SHOW CLUSTERS

Plumbing

Tests

Motivation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtabebe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aljoscha commented Jul 14, 2026 •

edited

Loading