ClickHouse Monitoring: Metrics That Matter

ClickHouse is unusually honest about its own state. Every counter, gauge, and log line the server produces is queryable with SQL, and most incidents - insert rejections, replica lag, OOM-killed queries - announce themselves in a system table long before users notice. The hard part isn't getting metrics out of ClickHouse. It's that the server exposes over a thousand of them, and maybe fifteen actually predict outages.

This post covers those fifteen or so: where each one lives, what healthy looks like, and where to set the alert. If you only build one ClickHouse dashboard, build it from this list.

Where the Metrics Live

ClickHouse ships its own instrumentation as tables. Three of them form the core, and it's worth being precise about what each one holds:

• `system.metrics` contains gauges - values that go up and down, sampled now: running queries, open TCP connections, bytes of tracked memory, busy background pool slots.
• `system.events` contains counters - monotonically increasing totals since server start: queries executed, rows inserted, inserts delayed, inserts rejected. You graph their rate, not their value.
• `system.asynchronous_metrics` contains values recomputed periodically in the background because they're expensive to calculate on demand: max parts per partition, worst replica delay, filesystem free space, OS-level CPU and memory. Our asynchronous metrics guide walks through the full table.

Behind those sit the log tables - `system.query_log` records every query with duration, rows read, and memory used; `system.part_log` records every part created, merged, or dropped; system.replication_queue shows pending replication work per table. The system tables overview maps the whole family.

You don't need an agent to export any of this. ClickHouse has an embedded Prometheus endpoint - a config block, not a plugin:

<prometheus>
    <endpoint>/metrics</endpoint>
    <port>9363</port>
    <metrics>true</metrics>
    <events>true</events>
    <asynchronous_metrics>true</asynchronous_metrics>
    <errors>true</errors>
</prometheus>

That serves all three metric tables in Prometheus format on port 9363. Two HTTP endpoints round out basic availability checks: `/ping` returns 200 when the server is up, and /replicas_status returns 503 when a replica has fallen too far behind - which makes it a ready-made load balancer health check.

The Shortlist: What to Alert On

Here is the reference table. Sources are the tables above; thresholds are starting points drawn from ClickHouse defaults and operational guides like Altinity's monitoring knowledge base, to be tuned against your own baseline.

Three failure domains produce nearly all real ClickHouse incidents: ingestion (parts and merges), replication (queues and Keeper), and resources (memory and disk). The next sections take them in that order.

Ingestion: Parts and the Merge Backlog

Every insert into a MergeTree table writes a new immutable part; background merges then combine small parts into larger ones. Monitoring ingestion health means monitoring one race: are merges keeping up with inserts? When they don't, part counts climb, and ClickHouse defends itself - by default it throttles inserts once a partition holds 1,000 active parts (`parts_to_delay_insert`) and rejects them outright at 3,000 (`parts_to_throw_insert`) with the notorious Too many parts error.

That's why MaxPartCountForPartition is the single best leading indicator in ClickHouse. It reports the worst partition on the server, it moves minutes to hours before inserts start failing, and a healthy steady state sits comfortably under 100. Alerting at 300 leaves room to act; waiting until 1,000 means your producers are already seeing latency.

Watch the throttling events themselves too. DelayedInserts and RejectedInserts in system.events should be flat lines at zero - any sustained rate is the defense mechanism already firing. And when part counts do climb, system.part_log tells you which side of the race broke: a flood of tiny NewPart entries means clients are inserting in batches that are too small (aim for tens of thousands of rows per insert, not hundreds), while sparse MergeParts entries mean merges are starved - check whether BackgroundMergesAndMutationsPoolTask is pinned at the pool ceiling. Computing your ingestion rate from part_log makes the comparison direct. We covered the mechanics of this race in Inside ClickHouse MergeTree.

Queries: Percentiles from query_log, Not Averages

Average query latency hides everything interesting - a dashboard average of 40 ms is fully compatible with a p99 of 30 seconds. system.query_log has per-query truth, so compute percentiles from it directly:

SELECT
    quantiles(0.5, 0.95, 0.99)(query_duration_ms) AS p50_p95_p99,
    count() AS queries,
    countIf(exception != '') AS failed
FROM system.query_log
WHERE type > 1
  AND event_time > now() - INTERVAL 15 MINUTE;

Alert on p99 drifting 2-3x above your baseline, and on FailedQuery rate separately - error spikes and latency spikes have different root causes. When latency does drift, query_log also carries the explanation: read_rows and read_bytes ballooning means a filter stopped pruning parts; memory_usage climbing toward max_memory_usage means aggregations are about to start spilling or failing. Sort by query_duration_ms over the incident window and the offending query pattern is usually obvious. Our query_log recipe collection has ready-made queries for exactly this.

One pitfall: query_log is itself a MergeTree table with a TTL. If you want week-over-week latency comparisons, confirm its retention covers the comparison window before you need it.

Replication and Keeper

Replicated tables coordinate through ZooKeeper or ClickHouse Keeper, and this dependency is the part of the system most teams under-monitor. Two asynchronous metrics summarize replica health server-wide: ReplicasMaxAbsoluteDelay (the worst replica lag, in seconds) and ReplicasMaxQueueSize (the deepest per-table replication queue). Lag past max_replica_delay_for_distributed_queries (default 300 s) gets replicas excluded from distributed reads, so 300 seconds is the natural alert line - and a queue that grows while num_tries climbs on one entry means a stuck task, with last_exception in system.replication_queue naming the reason.

The metric that means drop-everything is ReadonlyReplica in system.metrics. A nonzero value means tables have lost their coordination session and are refusing writes. Which is why Keeper itself belongs on the same dashboard: the `mntr` four-letter command (echo mntr | nc keeper-host 9181) reports zk_avg_latency, zk_outstanding_requests, and the leader/follower state, and Keeper has shipped its own Prometheus endpoint since ClickHouse 22.12. Keeper average latency above ~50 ms shows up downstream as slow inserts into replicated tables - Keeper participates in every part commit, so its latency is a floor under your write latency.

For resources, the pairing that matters is MemoryTracking against max_server_memory_usage (alert at 90%) and FilesystemMainPathAvailableBytes against real disk (alert at 20% free - merges need scratch space, and a disk-full ClickHouse fails in ugly, sometimes unrecoverable ways).

Key Takeaways

• ClickHouse monitoring is built in: system.metrics (gauges), system.events (counters), system.asynchronous_metrics (periodic), plus query_log, part_log, and replication_queue - all exportable via the embedded Prometheus endpoint on port 9363.
• MaxPartCountForPartition is the best leading indicator of ingestion trouble. Alert at 300; defaults throttle inserts at 1,000 parts per partition and reject at 3,000.
• Alert on query p99 from system.query_log, never on averages, and track FailedQuery rate separately.
• Replication health is three numbers: ReplicasMaxAbsoluteDelay (> 300 s), ReplicasMaxQueueSize (> 100), and ReadonlyReplica (> 0). Keeper latency above 50 ms is a write-path problem, not a sidebar.
• Thresholds here are starting points. Baseline your own workload for two weeks, then tighten.

The catch with all of the above is that thresholds are static and workloads aren't - a part count that's normal during a backfill is an incident at 4am on a Tuesday. That's the problem NeverBlink works on: it monitors ClickHouse (and your other databases) continuously, learns each cluster's baseline, and turns these system tables into diagnoses - which table, which setting, which query - instead of another wall of charts. The metrics above are where we'd tell you to start either way.