Prometheus Histograms for Spatial Endpoint Latency

← Back to Observability for Spatial Endpoints

This page covers the two decisions that determine whether a latency histogram is useful on a spatial API: where the bucket boundaries sit, and which labels are allowed to exist.

# Context & When to Use

A histogram answers percentile questions by counting observations into fixed buckets. That makes the bucket boundaries the resolution limit of every answer it can give: histogram_quantile interpolates within a bucket, so if 80 % of your traffic lands between the 25 ms and 100 ms boundaries, every percentile in that range is an interpolation across a four-fold span and the numbers are close to fiction.

Spatial APIs make this worse than most, because their distribution is not merely skewed but genuinely bimodal — small-envelope requests served from an index and cache in single-digit milliseconds, large-envelope requests that legitimately take seconds. Default buckets place almost no boundaries in the first mode and far too many in the gap between the two.

The second decision, labels, is where spatial APIs go wrong in a more dangerous way. The natural things to label by — the bounding box, the tile coordinates, the tenant — are all unbounded, and a metric with unbounded labels is how a monitoring system runs out of memory. Bounded proxies exist for all of them, and the magnitude bucketing from Observability for Spatial Endpoints is the important one.

# Runnable Implementation

from prometheus_client import Counter, Histogram

# Buckets chosen from a day of real durations, not from the library default.
# Dense through 5–250 ms where the indexed traffic lives, sparse in the tail.
SPATIAL_LATENCY = Histogram(
    "spatial_request_seconds",
    "End-to-end latency of a spatial request",
    labelnames=("operation", "magnitude", "status"),
    buckets=(
        0.005, 0.010, 0.020, 0.035, 0.050, 0.075,   # indexed reads
        0.100, 0.150, 0.250,                        # warm but working
        0.500, 1.0, 2.5, 5.0,                       # the honest tail
    ),
)

DB_LATENCY = Histogram(
    "spatial_db_seconds",
    "Time inside PostGIS, excluding pool wait and serialization",
    labelnames=("statement", "magnitude"),
    buckets=(0.002, 0.005, 0.010, 0.025, 0.050, 0.100, 0.250, 1.0, 3.0),
)

POOL_EXHAUSTED = Counter(
    "spatial_pool_exhausted_total",
    "Requests that waited more than 50 ms for a connection",
    labelnames=("operation",),
)

# Label allow-lists make the series budget a property of the code, not a hope.
OPERATIONS = frozenset({"bbox", "knn", "tile", "export", "other"})
MAGNITUDES = frozenset({"xs", "s", "m", "l", "xl", "na"})


def observe(operation: str, magnitude: str, status_code: int, seconds: float) -> None:
    """Record one request, coercing any unexpected label into a known bucket."""
    op = operation if operation in OPERATIONS else "other"
    mag = magnitude if magnitude in MAGNITUDES else "na"
    # Status CLASS, not code: 200 and 204 are the same story, 500 and 503 are not
    status = f"{status_code // 100}xx"
    SPATIAL_LATENCY.labels(operation=op, magnitude=mag, status=status).observe(seconds)

Coercing unknown values into other rather than passing them through is what keeps the series count fixed when someone adds a route and forgets to update the classifier.

# Key Parameters & Options

# Counting the series before shipping

The series budget is arithmetic, and it is worth doing before the metric reaches production rather than after the monitoring system falls over.

# Reading the result

A histogram is only worth its storage if someone looks at it, and the panels worth building are the ones that answer a question you would otherwise have to guess at.

# Deriving your own bucket boundaries

The buckets above suit the workload they were measured on. Yours will differ, and the derivation takes about ten minutes: log raw durations for a day, then place boundaries at the deciles of the observed distribution, rounding to friendly numbers.

import numpy as np

durations = np.loadtxt("durations_one_day.txt")       # seconds, one per line
deciles = np.percentile(durations, [10, 20, 30, 40, 50, 60, 70, 80, 90, 95, 99])
print([round(float(d), 3) for d in deciles])
# [0.006, 0.009, 0.013, 0.018, 0.024, 0.033, 0.048, 0.079, 0.186, 0.42, 2.31]

Round those to the nearest sensible value, add one boundary below the fastest observation and one above the slowest you care about, and stop. Ten to fourteen boundaries is the practical range: fewer and the percentiles blur, more and you are paying series count for resolution nobody reads. Re-derive after any change that shifts the distribution — a new index, a caching layer, a partitioning change — because buckets tuned to the old shape quietly stop resolving the new one.

# Gotchas & Failure Modes

• Measuring only successful requests. A try/except that observes on the happy path only removes exactly the slow, failing requests from the percentile. Observe in a finally.
• histogram_quantile over a counter that reset. Use rate() over the bucket counters, not the raw values, or a process restart produces a nonsensical spike.
• One dashboard panel for all operations. The aggregate p95 is dominated by whichever operation is most frequent. Break out by operation in every panel that matters.
• Buckets changed in place. Editing bucket boundaries makes historical data incomparable; the old series keep their old boundaries. Version the metric name if the change is significant.
• le labels drifting between instances. Every replica must use identical bucket definitions, or aggregation across them silently drops observations. Define the buckets in one module.
• Exposing /metrics publicly. It reveals route names, traffic volume and error rates. Bind it internally, as noted in Observability for Spatial Endpoints.

# Verification Snippet

from prometheus_client import REGISTRY


def test_series_budget_is_bounded(client, auth_headers):
    for bbox in ("-0.1,51.5,-0.09,51.51", "-10,40,10,60", "-180,-85,180,85"):
        for path in ("/v1/features", "/v1/nearest", "/v1/tiles/12/2048/1361.mvt"):
            client.get(path, params={"bbox": bbox}, headers=auth_headers)

    samples = [s for m in REGISTRY.collect()
               if m.name == "spatial_request_seconds" for s in m.samples]
    labels = {(s.labels.get("operation"), s.labels.get("magnitude"),
               s.labels.get("status")) for s in samples}
    assert len(labels) <= 5 * 6 * 5          # the declared budget
    assert all(o in OPERATIONS for o, _, _ in labels)

# p95 per operation — the panel that should exist before any other
histogram_quantile(0.95,
  sum by (le, operation) (rate(spatial_request_seconds_bucket[5m])))

# Small requests that got slow: the alert that catches real regressions
histogram_quantile(0.95,
  sum by (le) (rate(spatial_request_seconds_bucket{magnitude="xs"}[5m]))) > 0.1

# Related

• Observability for Spatial Endpoints — the signal design these metrics implement
• Instrumenting asyncpg Queries with OpenTelemetry — where high-cardinality context belongs instead
• Cost-Based Throttling for Expensive PostGIS Queries — the same magnitude estimate, used to charge rather than to measure

← Back to Observability for Spatial Endpoints