Operability & Dev Infrastructure

Metrics / Monitoring System

Scrape, store, query, and alert on operational metrics: a pull-based scraper with discovery, a time-series store, cardinality limits at ingest, RED/USE aggregation, and alerting that debounces flaps and missing data.

~30 min · intermediate

Problem & Requirements

Watch a fleet of services and machines: collect their metrics, keep the history, let operators query it, and page someone when something breaks. Each target exposes counters and gauges — request counts, error counts, latency histograms, CPU, memory — and the system samples them on an interval, stores the samples as time series, and evaluates alerting rules against them. The point of the whole thing is the alert at the end; storage and dashboards exist to make that alert correct and the debugging afterward possible.

The decision that shapes everything is who initiates collection. In a pull model the monitor holds the inventory of what should exist, discovers targets, and scrapes each one on a schedule, so a target it cannot reach is itself a signal — silence means down, not healthy. This is pull vs push, and it makes the monitor the authority on liveness rather than trusting targets to announce themselves. The samples land in time-series storage built for regular, append-only data, the series count is held down by cardinality control so labels can't multiply without bound, and queries roll raw series up into the signals operators actually read using RED and USE. The spine is that the monitor pulls, so it always knows what is supposed to be there.

Functional

scrape() pulls metrics from discovered targets on an interval and stores timestamped samples.
query(expr, range) aggregates series into rates, ratios, and quantiles.
alert(rule) evaluates conditions on a schedule and notifies when they hold.

Non-functional (back-of-envelope, single server)

Quantity	Target	What it forces
Scrape targets	~10k	Discovery plus parallel pull on an interval; the monitor owns the inventory
Scrape interval	15–60 s	Regular samples (compress well); resolution-vs-load tradeoff
Active series	10⁶–10⁷	Cardinality control; series count, not sample rate, is the limit
Ingest rate	~10⁵–10⁶ samples/s	Time-series storage — delta-of-delta + XOR, covered by the TSDB prototype
Query latency	sub-second per panel	Time-pruned columnar reads with aggregation pushed down
Alert evaluation	every 15–60 s, must not flap	A `for` duration to debounce; explicit handling of missing data
Notification	deduped, grouped, routed	One incident is not a thousand pages

The dangerous row is alert evaluation. Storing and querying numbers is the solved part — the Time-Series Database prototype already builds the engine, and cardinality control was its killer. The difficulty that surfaces only in production is the alert: a naive threshold flaps on noisy data, fires a storm when one cause trips fifty symptoms, and stays silent when the thing it watches disappears entirely (is the service down, or did the scrape fail?). Getting that right is build step six, and it is the line between a monitoring system and a pager nobody trusts.

Design

Six components, each tied to the principle or earlier prototype it builds on:

Scrape loop with discovery — the monitor pulls each discovered target's /metrics on an interval, timestamps centrally, and synthesizes an up series from whether the scrape succeeded. This is pull vs push on the pull side.
Exposition parsing — targets publish metrics as labelled name/value lines (the Prometheus/OpenMetrics text format); the scraper parses and attaches the target's labels.
Time-series store — samples route into a TSDB keyed by series id. This is time-series storage; the chunking, compression, and block internals are the Time-Series Database prototype and are not re-derived here.
Cardinality guard — caps on total series and on distinct values per label, applied at ingest, because discovered targets carry labels that can explode. This is cardinality control, the same guard the TSDB prototype builds.
Query and aggregation — a query layer computing rates, ratios, and quantiles and packaging them as the standard service and resource views. This is RED and USE.
Alerting and notification — rules evaluated on a schedule with debounce, missing-data handling, and grouping before anything pages a human.

The two named systems split on the spine decision. Prometheus is pull: a single server discovers targets through service discovery, scrapes them, stores locally in its own TSDB, evaluates rules in PromQL, and hands firing alerts to a separate Alertmanager for grouping and routing; exporters bridge things that can't expose metrics natively, and a push gateway handles the batch-job exception. Datadog is push: an agent on each host collects and pushes to a hosted, horizontally-scaled backend with tag-based metrics, which is more turnkey and shifts cardinality from a memory problem into a billing one — custom metrics are metered. Both face the cardinality wall; they differ in who initiates collection and in whether you operate the storage. The RED method (rate, errors, duration, from Tom Wilkie) and the USE method (utilization, saturation, errors, from Brendan Gregg) are the two aggregation conventions step five implements, and they are framework-independent.

Build it

Start with targets pushing their samples to an ingest endpoint that appends them, with a query that filters by series and range. It is correct and minimal. The failures are the case for pulling: a target that dies stops pushing and looks identical to a healthy but quiet one, so silence is ambiguous; the monitor has no inventory of what should exist; and the target supplies its own identity and timestamp, which it can get wrong. The next step inverts the direction.

Flip to pull. The monitor keeps a target list from service discovery, scrapes each target's /metrics on an interval, timestamps centrally, and records an up series — 1 if the scrape succeeded, 0 if it failed. This is pull vs push, and it resolves the ambiguity directly: failing to scrape a target is now a recorded signal you can alert on, and the monitor owns the inventory rather than waiting to be told. The failure it exposes is on the storage side — a flat list of samples neither compresses nor prunes, which collapses at a million samples a second.

Replace the flat list with a real time-series store. Each unique label set becomes a series id, and samples append into the engine that handles compression and time-pruned reads. This is time-series storage, and the engine internals — append-only columnar chunks, delta-of-delta timestamps, XOR float compression, time blocks — are exactly the Time-Series Database prototype, reused rather than rebuilt. The failure that follows is the one that prototype warned about: targets discovered in the wild carry labels, and a single runaway label spawns unbounded series.

Guard series creation. Cap the total number of active series and the distinct values any single label may take, and reject a new series that would breach either, so a label carrying a pod name, request path, or user id can't turn every scrape into millions of series. This is cardinality control, the same guard the Time-Series Database prototype builds, applied here at the scrape boundary where discovered labels enter. The failure that remains is one of usefulness: operators don't want raw counters, they want signals — request rate, error ratio, latency quantiles.

Add a query layer that turns raw series into the few signals people actually watch. The rate of a counter over a window, an error ratio, and a latency quantile compose into the RED view for a service — request Rate, Error rate, request Duration — and the parallel USE view for a resource is Utilization, Saturation, Errors. This is RED and USE, the conventions that make a dashboard legible at a glance. The failure left standing is that a dashboard is passive: nobody is watching it at 3am, and a naive threshold check flaps on noise and storms on correlated failures.

Make the alert trustworthy. Evaluate each rule on a schedule, but require the condition to hold for a for duration before firing so a brief spike doesn't page anyone, and treat missing data as a fireable condition in its own right — an up == 0 or a query that returns nothing means the target vanished, which is exactly when you most want to know. Then group and deduplicate notifications so one root cause that trips many rules pages once, not fifty times. This is the alerting correctness the requirements flagged as the real difficulty, and it sits on top of the query layer from step five.

import time

class Monitor:
    def __init__(self):
        self.samples = []                          # (series, ts, value)

    def ingest(self, series, value):               # target POSTs here
        self.samples.append((series, time.time(), value))

    def query(self, series, start, end):
        return [(t, v) for (s, t, v) in self.samples
                if s == series and start <= t < end]

Tradeoffs

Decision	What it buys	What it costs
Pull vs push (pull)	Central inventory, an `up` liveness signal, monitor-owned timestamps	Must reach every target; batch and behind-NAT jobs need a push gateway or agent
Time-series storage	Cheap regular-sample storage and time-pruned queries	The TSDB tradeoffs — irregular/out-of-order data and in-place edits stay hard
Cardinality control	Bounds memory and index at the source	Drops legitimate-looking series; in hosted systems the cap is literally the bill
RED / USE aggregation	The few signals operators need, legible at a glance	An aggregate can mask one bad instance among many healthy ones
Alerting with `for` + grouping	Fewer false pages, no storms, missing-data caught	Added latency to fire; a too-long `for` can mute a real fast incident

Scaling it up

A single server is the starting point, not the destination. Prometheus runs as one binary with local storage and scales out through functional sharding (split targets across servers), federation (a higher-level server scrapes aggregates from lower ones), and remote-write into a long-term, horizontally-scaled store such as Thanos, Cortex, or Mimir. Datadog is hosted and scaled by design. Either way, retention and downsampling become first-class — old data rolls to lower resolution and tiers to cheaper storage, the same aging-out the Time-Series Database prototype defers to its own scaling section.

Pull has a reachability gap that has to be designed around. The monitor must be able to open a connection to every target, which short-lived batch jobs, serverless functions, and hosts behind NAT defeat. The sanctioned exception is a push gateway that batch jobs push to and the monitor scrapes, plus agents for environments the monitor can't reach directly. These edge cases are where the clean pull model meets messy infrastructure, and getting them wrong reintroduces the silent-target ambiguity step two removed.

Alerting grows into a system of its own. Beyond debounce and grouping, production alerting adds inhibition (suppress a symptom alert when its known cause is already firing), silencing for planned maintenance, deduplication across replicated evaluators, and routing to the right on-call rotation — the work Alertmanager does. The bigger shift is from static thresholds to SLO-based burn-rate alerting from the Google SRE practice, where you alert on the rate at which an error budget is being consumed rather than on a raw value, which catches both fast and slow burns without the flapping a fixed threshold invites.

Cardinality control becomes governance rather than a single cap. At organization scale it means per-team series limits, relabeling rules that drop or aggregate offending labels at scrape time before they ever create series, and tooling that surfaces which label is exploding and who owns it. In hosted systems the same numbers are a cost-control problem because custom metrics are metered, so the cap is a budget conversation as much as a memory one. The flat limits from step four are the floor here, exactly as in the TSDB prototype.

Metrics alone are not observability. Correlating a metric spike with the logs and traces behind it — exemplars that link a latency bucket to a specific trace, OpenTelemetry as a common pipeline, OpenMetrics standardizing exposition — is where modern systems converge, and the alert is only the entry point to that investigation. From here the concrete follow-ons are a long-term-storage and federation prototype that builds remote-write, downsampling, and global query, an SLO burn-rate alerting prototype that replaces static thresholds with error-budget consumption, and a service-discovery and relabeling prototype that turns dynamic targets into stable, cardinality-safe series. Each extends this scrape-store-query-alert pipeline without re-treading the storage engine underneath it.

References

Prometheus documentation — pull model, exposition/OpenMetrics format, PromQL, and Alertmanager: https://prometheus.io/docs/
Beyer et al., Site Reliability Engineering (Google), alerting on SLOs and error-budget burn rate: https://sre.google/sre-book/monitoring-distributed-systems/
Wilkie, The RED Method — rate, errors, duration for services: https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/
Gregg, The USE Method — utilization, saturation, errors for resources: https://www.brendangregg.com/usemethod.html
Datadog documentation — agent-based push collection and custom-metric cardinality/metering: https://docs.datadoghq.com/metrics/
OpenMetrics / OpenTelemetry — exposition and pipeline standards for metrics, logs, and traces: https://openmetrics.io/ · https://opentelemetry.io/docs/
Thanos / Cortex / Grafana Mimir — long-term, horizontally-scaled storage behind Prometheus remote-write: https://thanos.io/ · https://grafana.com/oss/mimir/
Prior prototype: Time-Series Database — the storage engine this system stores into.
Foundations referenced inline: pull vs push · time-series storage · cardinality control · RED / USE method