Detecting Geofence Enumeration in the Audit Trail

← Back to Audit Logging for Location Data Access

This page shows how to find the account that is quietly rebuilding your dataset one bounding box at a time, using nothing but the envelopes already recorded in the audit table.

# Context & When to Use

Rate limiting stops the crude version of scraping. It does not stop the patient version: a client that requests one viewport-sized box every four seconds, stays comfortably inside every quota, and after a fortnight has retrieved every feature you have. Each individual request is indistinguishable from a legitimate map pan. What gives it away is the shape of the sequence — a sweep covers new ground on almost every call, while a human revisits the same few square kilometres over and over.

Because the audit trail stores each access as a geometry rather than a log line, that shape is directly queryable. Union a subject’s envelopes over a window and you have their coverage; divide coverage by request count and you have a ratio that separates the two behaviours cleanly. The trail design this depends on is described in Audit Logging for Location Data Access, and the throttle you reach for once a sweeper is found is in Rate Limiting Geofence & Tile Endpoints.

Run this as a scheduled report rather than in the request path. It is a detection control, not a gate, and it should be looking at hours of history rather than the last five seconds.

# Runnable Implementation

-- Coverage report: which subjects covered the most NEW ground per request?
WITH window_access AS (
    SELECT subject_id,
           envelope,
           row_count
    FROM   access_audit
    WHERE  occurred_at >= now() - interval '24 hours'
      AND  action = 'read'
      AND  envelope IS NOT NULL
),
coverage AS (
    SELECT subject_id,
           count(*)                                       AS requests,
           sum(row_count)                                 AS rows_returned,
           -- Total area asked for, counting overlaps repeatedly
           sum(ST_Area(envelope::geography)) / 1e6        AS requested_km2,
           -- Distinct ground actually covered, overlaps collapsed
           ST_Area(ST_Union(envelope)::geography) / 1e6   AS covered_km2
    FROM   window_access
    GROUP  BY subject_id
    HAVING count(*) >= 50            -- ignore casual traffic entirely
)
SELECT subject_id,
       requests,
       rows_returned,
       round(covered_km2::numeric, 1)                            AS covered_km2,
       round((covered_km2 / requests)::numeric, 3)               AS km2_per_request,
       -- 1.0 means every request was new ground; 0.05 means heavy revisiting
       round((covered_km2 / NULLIF(requested_km2, 0))::numeric, 3) AS novelty
FROM   coverage
ORDER  BY novelty DESC, covered_km2 DESC
LIMIT  25;

novelty is the discriminating column. It is the ratio of distinct ground covered to ground requested: a client that never repeats itself scores close to 1.0, while one that pans around a neighbourhood all day scores under 0.1 because its envelopes pile up on the same ground.

# Key Parameters & Options

# From detection to response

Finding a sweeper is the easy half. The response should be graduated, because the same signature is produced by a legitimate bulk consumer who simply picked the wrong endpoint for the job.

# Gotchas & Failure Modes

• ST_Union over a full day of envelopes is expensive. On a busy subject this can be tens of thousands of polygons. Aggregate hourly into a rollup table and union the rollups, or use ST_Union on a snapped grid rather than raw envelopes.
• Coarse envelopes flattening the signal. If the audit trail rounds envelopes to two decimal places, small adjacent requests collapse into the same box and novelty drops artificially. Two places is still fine at city scale; verify the rounding does not exceed a typical viewport.
• Tile traffic mixed with feature traffic. A map client legitimately requests hundreds of non-overlapping tiles per pan, which scores as pure novelty. Filter to layer values that return features, or exclude action = 'read' rows originating from tile routes.
• One subject, many API keys. A determined scraper spreads the sweep across credentials. Group by billing account as well as subject, and treat a set of accounts whose unions tile neatly together as one actor.
• Alerting on absolute area. A customer whose licence covers a whole country legitimately reads a whole country. Compare coverage against the scope in their token rather than against a global constant — the scope model is in Encoding Geofence Boundaries in JWT Scope Claims.
• No record of the decision. When throttling is applied, write it to the audit trail too. Otherwise next quarter nobody can explain why one account is slower than the rest.

# Choosing a threshold from your own traffic

There is no universal threshold, because the ratio that separates the two populations depends on how big a typical viewport is for your clients. Derive it empirically: run the report against a week of known-good traffic, plot the distribution of km2_per_request, and set the alert above the highest legitimate value with some headroom.

On most feature APIs the distribution is strongly bimodal, so the choice is easy — there is a wide empty band between the interactive population and anything automated. If your distribution has no such gap, that usually means tile traffic is mixed into the sample, or a legitimate bulk consumer is already using the feature endpoint as an export.

# Verification Snippet

-- Sanity check on known-good traffic: interactive users should score low
SELECT subject_id,
       count(*) AS requests,
       round((ST_Area(ST_Union(envelope)::geography) / 1e6
              / count(*))::numeric, 4) AS km2_per_request
FROM   access_audit
WHERE  occurred_at >= now() - interval '24 hours'
  AND  subject_id IN (SELECT subject_id FROM known_interactive_users)
GROUP  BY subject_id
ORDER  BY km2_per_request DESC
LIMIT  5;
--  subject_id | requests | km2_per_request
-- ------------+----------+-----------------
--  usr_8841   |      412 |          0.0752

# Simulate a sweep in staging and confirm the report flags it
python scripts/simulate_sweep.py --subject test_sweeper --tiles 400 --step 0.05
psql -f reports/coverage_novelty.sql | grep test_sweeper
# test_sweeper | 400 | 2840.0 | 7.100 | 0.981

# Related

• Audit Logging for Location Data Access — the envelope records this report reads
• Rate Limiting Geofence & Tile Endpoints — the throttle applied at escalation step two
• Cost-Based Throttling for Expensive PostGIS Queries — charging by area rather than by request

← Back to Audit Logging for Location Data Access