Streaming FlatGeobuf Responses from FastAPI

Serve a million features without buffering them in memory: FlatGeobuf's streamable layout, an async generator over a server-side cursor, and the headers that keep the download resumable.

← Back to GeoJSON vs GeoParquet Serialization

This page shows how to serve a very large feature collection as a FlatGeobuf stream, with constant memory on the server and a client that can start reading before the query has finished.

Context & When to Use

A bulk export endpoint that builds its response in memory has a hard ceiling. Serialising 800 000 polygons to GeoJSON produces roughly 1.4 GB of text; the server holds all of it, the client parses all of it before showing anything, and a worker that does this twice concurrently is out of memory. The usual mitigation — paginate the export — pushes the problem onto the consumer, who now has to stitch 400 pages together and handle a cursor that may drift.

FlatGeobuf is designed for exactly this shape. It is a flat binary format with a header describing the schema, followed by features that can be read one at a time, so both writer and reader work in constant memory. Combined with a server-side cursor in PostgreSQL, the whole path from disk to socket streams: no stage ever holds more than a chunk.

Reach for it when a single response legitimately contains more features than a client should hold in memory at once, and when the consumer is a GIS tool or data pipeline rather than a browser. For interactive map traffic, vector tiles are the better answer, and for analytical consumers the columnar layout compared in GeoJSON vs GeoParquet Serialization may serve better still.

Runnable Implementation

from typing import Annotated, AsyncIterator

import asyncpg
from fastapi import APIRouter, Depends, Query
from fastapi.responses import StreamingResponse

router = APIRouter(prefix="/v1/exports", tags=["exports"])

CHUNK = 5_000          # features encoded per yield

EXPORT_SQL = """
SELECT id, layer, category_code, ST_AsBinary(geom) AS wkb
FROM   features
WHERE  geom && ST_MakeEnvelope($1, $2, $3, $4, 4326)
ORDER  BY id
"""


async def flatgeobuf_stream(
    pool: asyncpg.Pool, bbox: tuple[float, float, float, float]
) -> AsyncIterator[bytes]:
    """Yield a FlatGeobuf file in chunks, holding at most CHUNK features."""
    writer = FlatGeobufWriter(          # thin wrapper over the fgb encoder
        geometry_type="MultiPolygon",
        columns=[("id", "long"), ("layer", "string"), ("category_code", "int")],
        crs=4326,
    )
    yield writer.header()               # magic bytes + schema, before any feature

    conn = await pool.acquire()
    tx = conn.transaction()
    await tx.start()                    # a cursor requires a transaction
    try:
        cursor = await conn.cursor(EXPORT_SQL, *bbox)
        buffer = bytearray()
        while True:
            rows = await cursor.fetch(CHUNK)
            if not rows:
                break
            for r in rows:
                buffer += writer.feature(
                    wkb=r["wkb"],
                    values=(r["id"], r["layer"], r["category_code"]),
                )
            yield bytes(buffer)
            buffer.clear()              # constant memory: one chunk at a time
    finally:
        # Without this, an aborted download leaks a connection AND a transaction
        await tx.rollback()
        await pool.release(conn)


@router.get("/features.fgb")
async def export_features(
    bbox: Annotated[str, Query(description="minx,miny,maxx,maxy in EPSG:4326")],
    pool: asyncpg.Pool = Depends(get_pool),
) -> StreamingResponse:
    minx, miny, maxx, maxy = (float(v) for v in bbox.split(","))
    return StreamingResponse(
        flatgeobuf_stream(pool, (minx, miny, maxx, maxy)),
        media_type="application/vnd.flatgeobuf",
        headers={
            "Content-Disposition": 'attachment; filename="features.fgb"',
            # No Content-Length is possible: the size is unknown until the end
            "X-Accel-Buffering": "no",      # stop nginx buffering the whole body
        },
    )

The finally block is the load-bearing part. A client that closes the connection at 40 % — a user pressing cancel, a proxy timing out — cancels the generator, and without explicit cleanup the transaction stays open and the connection never returns to the pool.

Server memory during an 800 000 feature exportTwo memory curves over the life of one export. The buffered approach climbs steadily as features accumulate, peaking at 1.4 gigabytes just before the response is sent, then dropping to zero. The streamed approach stays flat at about 40 megabytes throughout, one chunk at a time. A dashed line marks the container memory limit at 1 gigabyte, which the buffered curve crosses at around 70 percent completion — the point at which the worker is killed.Server memory during one 800 000-feature export1.5 GB750 MB0container limit 1 GBOOM-killed at 70 %streamed: flat at ~40 MBstart50 %completeThe streamed curve is flat because the chunk buffer is cleared after every yield — memory is a function ofchunk size

Key Parameters & Options

SettingValueEffect
CHUNK5 000 featuresLarger chunks reduce yields but raise peak memory and block the loop longer
cursor.fetch()server-side cursorWithout it asyncpg materialises the whole result set
Transactionexplicit, with finallyA cursor needs one; a leak here starves the pool
X-Accel-Buffering: norequired behind nginxOtherwise the proxy buffers the entire body and the streaming is lost
Content-DispositionattachmentMakes browsers save rather than attempt to render binary
Content-LengthomittedUnknowable mid-stream; chunked encoding instead

Chunk size is the one number worth tuning. Too small and the overhead of yielding dominates; too large and each encode blocks the event loop long enough to delay other requests. Five thousand simple features is a good starting point — measure the encode time per chunk and keep it under about 20 ms.

What streaming actually buys

Three ways to serve the same 800 000 featuresThree approaches compared on three measures. Buffered GeoJSON produces a 1420 megabyte payload, a time to first byte of 96 seconds and peak memory of 1.4 gigabytes. Streamed GeoJSON produces the same 1420 megabytes but a time to first byte of 0.4 seconds and peak memory of 45 megabytes. Streamed FlatGeobuf produces 268 megabytes, a time to first byte of 0.3 seconds and peak memory of 41 megabytes. The format change cuts size fivefold; the streaming change cuts time to first byte and memory by two orders of magnitude.Same query, three delivery choicesApproachPayload MBFirst bytePeak RAMbuffered GeoJSON1 42096 s1.4 GBstreamed GeoJSON1 4200.4 s45 MBstreamed FlatGeobuf268 MB0.3 s41 MBThe two changes are independent: streaming fixes memory and latency, the format fixes size. Doing only onestill leaves a real problem — a streamed 1.4 GB of JSON still costs the client minutes of parsing.Measured on 800 000 building polygons, average 34 vertices, five attributes.

Where the bytes come from

Knowing the layout helps when a reader rejects the output, because the failure is almost always in the header rather than in the features.

FlatGeobuf layout, as written by the streamThe file begins with eight magic bytes identifying the format and version. A header follows, declaring the geometry type, the coordinate reference system and the column schema, and it must be written before any feature. A spatial index section is optional and is omitted when streaming because it requires knowing every feature's extent in advance. The remaining bytes are features written one after another, each self-contained, which is what allows a reader to consume them incrementally.What the generator emits, in ordermagic8 bytesheadergeometry type · CRS · columnsindexomitted when streamingfeature 1feature 2feature 3…first yield — before any row is readone yield per CHUNK featuresBecause the index needs every extent up front, a streamed file has none — readers fall back to asequential scan, which is exactly what a streaming consumer was going to do anyway.If a GIS tool rejects the output, check the header first: a wrong declared geometry type is the usual cause.

Gotchas & Failure Modes

  • A proxy that buffers. nginx and several managed load balancers buffer responses by default, which silently converts a streamed response back into a buffered one — on the proxy’s memory instead of yours. X-Accel-Buffering: no handles nginx; check the equivalent for your edge.
  • Leaked connections on client cancel. Without finally, every cancelled download costs one connection and one open transaction until the server restarts. This exhausts the pool faster than any query, as described in Connection Pooling & PgBouncer Setup.
  • A long-lived transaction blocking vacuum. A ten-minute export holds a snapshot for ten minutes, during which dead tuples on the whole database cannot be reclaimed. Cap the export size, or run exports against a replica.
  • Errors after the first byte. Once the response has started, the status code is already 200 and there is no way to signal failure except by truncating. Validate everything — parameters, permissions, geometry type — before the first yield.
  • Mixed geometry types. FlatGeobuf’s header declares one geometry type. A query returning both polygons and points needs Unknown as the declared type, which some readers handle poorly. Filter by type, or split the export.
  • No resumability. A failed download at 90 % starts again from zero. For very large exports, write to object storage and return a pre-signed URL, so the client’s HTTP range support does the resuming — the pattern in Async Bulk Uploads with Celery applied in reverse.

Verification Snippet

# First byte should arrive in well under a second, long before completion
curl -s -o /dev/null -w 'first_byte=%{time_starttransfer}s total=%{time_total}s size=%{size_download}\n' \
  "http://localhost:8000/v1/exports/features.fgb?bbox=-1,50,1,52"
# first_byte=0.31s total=41.7s size=281018368

# The file must be readable by a standard GIS tool
ogrinfo -so -al /vsicurl/"http://localhost:8000/v1/exports/features.fgb?bbox=-1,50,1,52"
# Feature Count: 798412
# Geometry: Multi Polygon
async def test_memory_stays_flat(pool, monkeypatch):
    import tracemalloc
    tracemalloc.start()
    total = 0
    async for chunk in flatgeobuf_stream(pool, (-1, 50, 1, 52)):
        total += len(chunk)
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    assert total > 10_000_000          # a real export happened
    assert peak < 100 * 1024 * 1024    # and memory never grew with it

← Back to GeoJSON vs GeoParquet Serialization