Observability for Spatial Endpoints

← Back to Deploying and Operating Geospatial APIs

A generic HTTP dashboard tells you a spatial API is slow. It does not tell you that the slowness is confined to requests whose bounding box exceeds two square degrees, that those requests started missing the GiST index after last week’s ANALYZE, or that the index no longer fits in the buffer cache. Spatial workloads fail in ways that route-level metrics average away, and the diagnosis almost always lives in the shape of the query rather than in the endpoint that issued it.

This page sets out an instrumentation layer built around that fact: metrics labelled by spatial operation and magnitude, traces that carry the query envelope and row count, sampled plans for the slow tail, and the four database signals that go bad before the API does. It assumes the deployment model from Containerizing PostGIS & FastAPI and complements the tuning work in Query Plan Analysis & Index Tuning.

# Prerequisites & Environment

FastAPI 0.110+, asyncpg 0.29, opentelemetry-sdk 1.24+ with the OTLP exporter, and prometheus-client 0.20+. On the database side, enable the two extensions that make query-level attribution possible:

-- postgresql.conf
-- shared_preload_libraries = 'pg_stat_statements'
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
CREATE EXTENSION IF NOT EXISTS pgstattuple;   -- for index bloat measurement

-- Log the statements that matter, not all of them
ALTER SYSTEM SET log_min_duration_statement = '500ms';
ALTER SYSTEM SET auto_explain.log_min_duration = '2s';
ALTER SYSTEM SET auto_explain.log_analyze = on;
SELECT pg_reload_conf();

auto_explain is worth the small overhead on a spatial workload: the plan for a query that took four seconds at 02:00 is otherwise unrecoverable, because re-running it during business hours produces a different plan against a warm cache.

# The four signals, and what each one catches

# Step-by-Step Implementation

# 1. Label metrics by operation and magnitude, never by geometry

The label set is the whole design. operation separates workload shapes that have genuinely different latency distributions; magnitude is a bucketed proxy for how much work the request asked for. Both are bounded, so the time-series count stays fixed.

from prometheus_client import Histogram, Counter

SPATIAL_LATENCY = Histogram(
    "spatial_request_seconds",
    "End-to-end latency of a spatial request",
    labelnames=("operation", "magnitude", "status"),
    # Buckets chosen around the real distribution, not the defaults
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
)

ROWS_RETURNED = Histogram(
    "spatial_rows_returned",
    "Rows returned per spatial query",
    labelnames=("operation",),
    buckets=(1, 10, 100, 1_000, 10_000, 100_000),
)

INDEX_FALLBACK = Counter(
    "spatial_seq_scan_total",
    "Queries observed falling back to a sequential scan",
    labelnames=("operation",),
)


def magnitude_bucket(area_deg2: float) -> str:
    """Bounded label: how much of the world the request asked about."""
    if area_deg2 < 0.01:
        return "xs"        # a few city blocks
    if area_deg2 < 1:
        return "s"         # a city
    if area_deg2 < 25:
        return "m"         # a region
    if area_deg2 < 500:
        return "l"         # a country
    return "xl"            # continental or unbounded

An xl request that takes eight seconds is working as designed; an xs request that takes eight seconds is an incident. Without the magnitude label those two are the same data point, which is why undifferentiated dashboards never catch the second case.

# 2. Span the database call with spatial context

The request span is not enough — almost all the time is inside one statement, and the interesting attributes are its parameters.

import json
from contextlib import asynccontextmanager

import asyncpg
from opentelemetry import trace

tracer = trace.get_tracer("geospatial-api")


@asynccontextmanager
async def spatial_query(
    conn: asyncpg.Connection,
    name: str,
    *,
    operation: str,
    envelope_deg2: float | None = None,
):
    """Wrap one statement in a span carrying the context needed to diagnose it."""
    with tracer.start_as_current_span(f"db.{name}") as span:
        span.set_attribute("db.system", "postgresql")
        span.set_attribute("db.statement_name", name)
        span.set_attribute("geo.operation", operation)
        if envelope_deg2 is not None:
            span.set_attribute("geo.envelope_deg2", round(envelope_deg2, 4))
            span.set_attribute("geo.magnitude", magnitude_bucket(envelope_deg2))
        yield span


async def fetch_features(conn, sql: str, args: tuple, *, operation: str, area: float):
    async with spatial_query(conn, "features_bbox", operation=operation,
                             envelope_deg2=area) as span:
        rows = await conn.fetch(sql, *args)
        span.set_attribute("db.rows", len(rows))
        ROWS_RETURNED.labels(operation=operation).observe(len(rows))
        return rows

Recording the row count next to the duration is what makes the trace self-explaining: 900 ms for 40 000 rows is arithmetic, 900 ms for 12 rows is a missing index.

# 3. Sample a plan for the slow tail

import asyncio
import time

PLAN_BUDGET_S = 1.0
_plan_tokens = 6          # at most six sampled plans per minute
_plan_window = 0.0


async def maybe_capture_plan(pool, sql: str, args: tuple, elapsed: float, span) -> None:
    """Attach an EXPLAIN plan to the span when a read blew its budget."""
    global _plan_tokens, _plan_window
    if elapsed < PLAN_BUDGET_S:
        return
    now = time.monotonic()
    if now - _plan_window > 60:
        _plan_tokens, _plan_window = 6, now
    if _plan_tokens <= 0:
        span.set_attribute("geo.plan_sampled", False)
        return
    _plan_tokens -= 1

    # Separate connection: never re-enter the one serving the request
    async with pool.acquire() as diag:
        await diag.execute("SET LOCAL statement_timeout = '5s'")
        plan = await diag.fetchval(f"EXPLAIN (FORMAT JSON, BUFFERS, ANALYZE) {sql}", *args)
    span.set_attribute("geo.plan_sampled", True)
    span.set_attribute("db.plan", json.dumps(plan)[:8000])

    # A plan that never touches the GiST index is worth counting, not just tracing
    if "Seq Scan" in str(plan):
        INDEX_FALLBACK.labels(operation=span.attributes.get("geo.operation", "unknown")).inc()

ANALYZE here executes the statement a second time. That is acceptable for a read that is already slow and rate-limited to six per minute; it is never acceptable for a mutation, so keep this path on the read routes only.

# 4. Scrape the database signals

-- Index versus sequential access on the geometry tables
SELECT relname,
       seq_scan,
       idx_scan,
       round(100.0 * seq_scan / NULLIF(seq_scan + idx_scan, 0), 2) AS pct_seq,
       n_live_tup,
       n_dead_tup,
       round(100.0 * n_dead_tup / NULLIF(n_live_tup, 0), 2)        AS pct_dead,
       last_autovacuum
FROM   pg_stat_user_tables
WHERE  relname IN ('features', 'positions', 'parcels')
ORDER  BY seq_scan DESC;

-- Is the GiST index still served from cache?
SELECT indexrelname,
       pg_size_pretty(pg_relation_size(indexrelid))                AS size,
       idx_blks_read,
       idx_blks_hit,
       round(idx_blks_hit::numeric
             / NULLIF(idx_blks_hit + idx_blks_read, 0), 4)         AS hit_ratio
FROM   pg_statio_user_indexes
WHERE  indexrelname LIKE '%_gix'
ORDER  BY hit_ratio ASC;

# 5. Read the histogram as two populations, not one

The payoff for labelling by magnitude arrives the first time you look at a latency distribution and see the shape rather than a single number. A healthy spatial service produces a distinctly bimodal picture: a tall, narrow mode for small-envelope requests served from the index and the buffer cache, and a low, wide mode for the large-envelope requests that genuinely have work to do. Both are fine. What is not fine is mass appearing between them, or the small-envelope mode drifting right — that is the signature of a plan regression or a cache that has stopped holding the working set.

Averaging those two populations produces a number that describes neither, and a single alert threshold on it either fires constantly or never fires at all. Alert per magnitude bucket instead: the xs bucket gets a tight budget measured in tens of milliseconds, the xl bucket gets seconds, and each is compared against its own baseline.

# Production Code Example

The middleware that ties operation, magnitude, latency and trace together:

import time
from typing import Callable

from fastapi import Request, Response
from opentelemetry import trace
from starlette.middleware.base import BaseHTTPMiddleware

OPERATION_BY_PREFIX = {
    "/v1/features":  "bbox",
    "/v1/nearest":   "knn",
    "/v1/tiles":     "tile",
    "/v1/exports":   "export",
}


def classify(path: str) -> str:
    for prefix, op in OPERATION_BY_PREFIX.items():
        if path.startswith(prefix):
            return op
    return "other"


def envelope_area(bbox: str | None) -> float | None:
    if not bbox:
        return None
    try:
        minx, miny, maxx, maxy = (float(v) for v in bbox.split(","))
    except ValueError:
        return None
    return abs(maxx - minx) * abs(maxy - miny)


class SpatialTelemetryMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next: Callable) -> Response:
        operation = classify(request.url.path)
        if operation == "other":
            return await call_next(request)

        area = envelope_area(request.query_params.get("bbox"))
        magnitude = magnitude_bucket(area) if area is not None else "na"

        span = trace.get_current_span()
        span.set_attribute("geo.operation", operation)
        span.set_attribute("geo.magnitude", magnitude)
        if area is not None:
            span.set_attribute("geo.envelope_deg2", round(area, 4))

        started = time.perf_counter()
        status = "500"
        try:
            response = await call_next(request)
            status = str(response.status_code)
            return response
        finally:
            elapsed = time.perf_counter() - started
            SPATIAL_LATENCY.labels(
                operation=operation, magnitude=magnitude, status=status
            ).observe(elapsed)
            span.set_attribute("geo.elapsed_ms", round(elapsed * 1000, 1))

Note the finally: a request that raises must still be measured, or the p99 quietly excludes exactly the requests that failed. This is the most common instrumentation bug in an otherwise well-monitored service.

# Verification & Testing

Assert on the label set rather than on the numbers — labels are the part that breaks silently.

from prometheus_client import REGISTRY


def test_latency_labels_are_bounded(client, auth_headers):
    for bbox in ("-0.1,51.5,-0.09,51.51", "-10,40,10,60", "-180,-85,180,85"):
        client.get("/v1/features", params={"bbox": bbox}, headers=auth_headers)

    samples = [
        s for m in REGISTRY.collect() if m.name == "spatial_request_seconds"
        for s in m.samples
    ]
    magnitudes = {s.labels["magnitude"] for s in samples}
    assert magnitudes <= {"xs", "s", "m", "l", "xl", "na"}
    # No coordinate ever reaches a label
    assert not any("." in s.labels.get("operation", "") for s in samples)

And confirm end to end that a slow request produces a plan on its span:

# Force a deliberately expensive request, then look for the sampled plan
curl -s "http://localhost:8000/v1/features?bbox=-180,-85,180,85&limit=5000" > /dev/null
# In the trace backend, the span db.features_bbox should carry:
#   geo.magnitude=xl  geo.plan_sampled=true  db.plan={"Plan":{...}}

# Failure Modes & Edge Cases

1. Cardinality explosion from a well-meaning label. Adding tenant_id or layer to a histogram multiplies the series count by the number of tenants. Put them on spans. If a per-tenant metric is genuinely required, use a separate counter with a hard allow-list of top tenants and an other bucket.
2. Percentiles averaged across operations. A dashboard showing one p95 for the whole API is dominated by whichever operation is most frequent. Always break it down by operation; the aggregate is only useful as an SLO headline.
3. Histogram buckets left at defaults. The Prometheus defaults top out at 10 s and are dense around 0.5 s, which is the wrong resolution for a workload whose interesting range is 5–200 ms. Choose buckets from your own distribution or every quantile below p90 reads as the same number.
4. EXPLAIN ANALYZE on the request connection. Re-entering the connection mid-request deadlocks under some pool configurations and doubles the work in all of them. Always use a separate connection, and cap the sample rate.
5. Traces without the row count. A span that says “912 ms” and nothing else cannot distinguish a big result from a bad plan. Row count is the cheapest attribute with the highest diagnostic value.
6. Alerting on latency alone. By the time p95 doubles, the cache is already cold and users have noticed. The hit ratio and dead-tuple signals move days earlier — see the chart above.
7. Metrics endpoint exposed publicly. /metrics reveals internal route names, tenant counts and traffic volumes. Bind it to an internal listener or require authentication, in line with the controls in Securing Geospatial APIs.
8. Sampling that discards the slow tail. Head-based trace sampling at 1 % will almost never keep the slow requests. Use tail-based sampling, or force sampled=true on any span that exceeds its budget.

# Performance Notes

The instrumentation described here costs about 0.15 ms per request: roughly 40 µs for the histogram observations, 60 µs for span creation and attributes, and the rest in context propagation. That is under 1 % of a typical 25 ms bounding box request and invisible next to a 400 ms tile.

The exception is plan sampling. An EXPLAIN (ANALYZE) re-runs the statement, so a 4-second query costs another 4 seconds of database time. Six per minute is a deliberate cap: enough to catch a regression within one alert window, few enough that a widespread slowdown does not turn the diagnostics into the outage. When a slow query also holds a connection, the interaction with pool sizing described in Connection Pooling & PgBouncer Setup is what decides whether sampling is safe at all.

Retention matters for the spatial signals specifically. Index bloat and cache-hit trends are only legible over weeks, so keep those series at a coarse resolution for 90 days even if request metrics roll off at 14.

# Related

• CI/CD Pipelines for Spatial APIs — catching plan regressions before they reach production
• Containerizing PostGIS & FastAPI — where the exporters and collectors live in the deployment
• Query Plan Analysis & Index Tuning — reading the plans this layer captures
• Connection Pooling & PgBouncer Setup — the saturation signal behind most latency cliffs
• Audit Logging for Location Data Access — the security counterpart to this operational trail

← Back to Deploying and Operating Geospatial APIs