Health Checks and Readiness Probes for PostGIS APIs

← Back to Containerizing PostGIS & FastAPI

This page covers the difference between liveness and readiness for a spatial API, and why conflating them turns a short database hiccup into a full outage.

# Context & When to Use

Kubernetes asks two different questions and takes two very different actions. Liveness asks “is this process broken beyond recovery?” and the answer being no results in the container being killed. Readiness asks “can this instance serve traffic right now?” and the answer being no results in it being taken out of the load balancer, still running, still able to come back.

Wiring both to the same handler — one that opens a connection and runs a query — collapses the distinction. When the database becomes briefly unreachable, every replica fails liveness at the same moment and the orchestrator restarts all of them. They come back with empty connection pools, cold PROJ pipeline caches and no warm statement plans, into a database that has just recovered and is now handling a thundering herd. A thirty-second blip becomes several minutes of degraded service, caused entirely by the health check.

The correct split is simple: liveness tests only what a restart could fix, readiness tests everything needed to serve. For a PostGIS API that also means being careful about what the readiness check asks the database, because it runs on every replica every few seconds — the same cost discipline as anything else on the hot path described in Containerizing PostGIS & FastAPI.

# Runnable Implementation

import asyncio
import time
from typing import Any

import asyncpg
from fastapi import APIRouter, Depends, Response, status

router = APIRouter(tags=["ops"])

STARTUP_CHECKS: dict[str, Any] = {}      # filled once, at startup


@router.get("/healthz", include_in_schema=False)
async def liveness() -> dict[str, str]:
    """Liveness: is THIS PROCESS alive? No I/O, no dependencies, no database."""
    return {"status": "ok"}


@router.get("/readyz", include_in_schema=False)
async def readiness(
    response: Response,
    pool: asyncpg.Pool = Depends(get_pool),
) -> dict[str, Any]:
    """Readiness: can this instance serve a request right now?"""
    checks: dict[str, Any] = {}
    healthy = True

    started = time.perf_counter()
    try:
        # A trivial query, with a hard timeout: the probe must never hang
        async with asyncio.timeout(2):
            async with pool.acquire() as conn:
                await conn.fetchval("SELECT 1")
        checks["database"] = {"ok": True,
                              "ms": round((time.perf_counter() - started) * 1000, 1)}
    except (asyncio.TimeoutError, asyncpg.PostgresError, OSError) as exc:
        checks["database"] = {"ok": False, "error": exc.__class__.__name__}
        healthy = False

    # Pool saturation is a readiness concern: a full pool cannot serve
    checks["pool"] = {"size": pool.get_size(), "idle": pool.get_idle_size()}
    if pool.get_idle_size() == 0 and pool.get_size() >= pool.get_max_size():
        checks["pool"]["ok"] = False
        healthy = False

    # Startup facts, reported but not re-tested
    checks["postgis"] = STARTUP_CHECKS.get("postgis", {"ok": False})

    if not healthy:
        response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
    return {"status": "ready" if healthy else "not_ready", "checks": checks}


async def verify_at_startup(pool: asyncpg.Pool) -> None:
    """Run the expensive, one-off checks once — and fail the deploy if they fail."""
    async with pool.acquire() as conn:
        version = await conn.fetchval("SELECT postgis_full_version()")
        # A round trip through PROJ proves the grid-shift data is present
        drift_m = await conn.fetchval(
            """
            SELECT ST_Distance(
                     ST_SetSRID(ST_MakePoint(-0.1276, 51.5072), 4326)::geography,
                     ST_Transform(ST_Transform(
                       ST_SetSRID(ST_MakePoint(-0.1276, 51.5072), 4326), 27700),
                       4326)::geography)
            """
        )
    if drift_m is None or drift_m > 0.001:
        raise RuntimeError(f"PROJ grid-shift data missing: round-trip drift {drift_m} m")
    STARTUP_CHECKS["postgis"] = {"ok": True, "version": version,
                                 "proj_round_trip_m": round(drift_m, 6)}

The startup check is where the strict assertions belong. A container built without proj-data transforms national grids at metre rather than centimetre accuracy — the failure described in Coordinate Reference Systems & SRID Handling — and it should stop the rollout, not degrade quietly in production.

# Key Parameters & Options

livenessProbe:
  httpGet: { path: /healthz, port: 8000 }
  periodSeconds: 30
  failureThreshold: 5          # generous: only a truly wedged process should die
readinessProbe:
  httpGet: { path: /readyz, port: 8000 }
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3          # tight: stop sending traffic quickly
startupProbe:
  httpGet: { path: /readyz, port: 8000 }
  failureThreshold: 30         # allow 5 minutes for migrations on first boot
  periodSeconds: 10

# What the probe costs at scale

# Which dependency belongs in readiness

The last judgement call is which dependencies get a vote. The test is whether the instance can serve a useful response without the dependency — not whether everything is nominal.

# Gotchas & Failure Modes

• Liveness and readiness pointing at the same handler. The most common configuration and the most damaging. One line of YAML separates a blip from an outage.
• No timeout inside the readiness handler. A probe that hangs on a wedged connection never returns, the kubelet times out, and the instance is marked unready for a reason nobody can see in the logs. Use an explicit internal timeout shorter than the probe’s.
• Readiness that ignores the pool. An instance whose pool is fully saturated will accept a request and queue it behind everything else. Reporting unready sheds load to healthier replicas — the saturation signal from Connection Pooling & PgBouncer Setup.
• A startupProbe that is too strict. First boot may run migrations. Without a generous startup probe the container is killed mid-migration, which is considerably worse than a slow deploy.
• Probes exposed publicly. /readyz reveals pool sizes, versions and error classes. Bind them to an internal port or restrict by network policy.
• Checking a dependency the API does not need. If the service can serve tiles from cache without Redis, a failed Redis check should not mark it unready. Readiness means “can serve”, not “everything is perfect”.

One further nuance is worth stating explicitly: readiness should be allowed to flap. An instance that goes unready for twenty seconds during a connection storm and then recovers has done exactly what the mechanism is for, and treating that as an incident encourages people to loosen the check until it never fires. Alert on the proportion of replicas unready at once, not on any single replica flipping — one replica shedding load is the system working, and all of them doing it at once is the thing worth waking someone for.

# Verification Snippet

# Liveness must answer even with the database stopped
docker compose stop db
curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/healthz   # 200
curl -s -o /dev/null -w '%{http_code}\n' localhost:8000/readyz    # 503
docker compose start db
sleep 5
curl -s localhost:8000/readyz | jq '{status, db: .checks.database.ok}'
# {"status":"ready","db":true}

async def test_liveness_does_not_touch_the_database(client, broken_pool):
    r = await client.get("/healthz")
    assert r.status_code == 200          # no dependency, no failure

async def test_readiness_reports_503_when_pool_is_dead(client, broken_pool):
    r = await client.get("/readyz")
    assert r.status_code == 503
    assert r.json()["checks"]["database"]["ok"] is False

# Related

• Containerizing PostGIS & FastAPI — the deployment these probes belong to
• Pinning PostGIS Versions in Production Images — what the startup check verifies
• Observability for Spatial Endpoints — the signals that explain why readiness flipped

← Back to Containerizing PostGIS & FastAPI