Compliant Web Scraping

Exposing scraper metrics with the Python prometheus_client #

Before any Grafana panel can plot crawl health, your scraper has to publish numbers Prometheus can scrape, and this page covers exactly that instrumentation layer using the official Python prometheus_client. It sits under Crawl Observability with Prometheus and Grafana within the Pipeline Storage, Deduplication & Monitoring section, and stays deliberately narrow: which metric type to reach for, how to label without wrecking cardinality, and how to stand up a /metrics endpoint that a Prometheus server can poll. The companion guide on building a Grafana dashboard for crawl health consumes precisely the series we define here.

Problem Framing #

Scrapers fail in ways that are quiet until they are catastrophic: latency creeps up as a target adds bot defenses, error rates climb after a markup change, and retry loops burn budget without anyone noticing. Instrumenting the fetch loop with counters, histograms, and gauges turns those silent failures into queryable time series long before they page you at 3 a.m.

Collection modes for a crawler processA forking crawler needs the multiprocess collector, not the default registry.Collection modes for a crawler processModeFitsCaveatHTTP endpoint scrapedLong-lived workersNeeds a portMultiprocess directoryForked worker poolsGauge semanticsPush gatewayShort batch jobsStale seriesSidecar exporterNon-Python workersExtra moving part
A forking crawler needs the multiprocess collector, not the default registry.

The decision that trips up most engineers is not whether to add metrics but how to shape their labels. Prometheus creates one independent time series per unique label-value combination, so a single ill-chosen label — say, the full request URL — can generate millions of series and knock over your monitoring stack faster than the scraper itself ever could. The rest of this page is about instrumenting the right signals with disciplined, low-cardinality labels.

Step-by-Step Implementation #

  1. Pick the right metric type per signal. Use a Counter for monotonically increasing totals (requests sent, items parsed, retries), a Histogram for distributions you will percentile later (fetch latency, response size), and a Gauge for values that go up and down (in-flight requests, frontier queue depth).
  2. Define metrics once, at module scope. Metric objects register into a global collector on creation, so instantiate them at import time — never inside the request loop, or you will hit Duplicated timeseries errors.
  3. Choose bounded labels. Label by host and coarse status class only. Never label by URL, user ID, or timestamp.
  4. Instrument the fetch path with .labels(...).inc(), .observe(), and gauge context managers.
  5. Expose /metrics via start_http_server on a dedicated port so Prometheus can scrape it.
Instrumenting one requestLabelling by status class, not raw status, keeps the series count sane.Instrumenting one request1Start a timerbefore the fetch2Increment therequest counter3Observe theduration4Label by hostand status class
Labelling by status class, not raw status, keeps the series count sane.
import time
import random
from urllib.parse import urlsplit
from prometheus_client import Counter, Histogram, Gauge, start_http_server
import httpx

# --- Metric definitions (module scope, registered once) ---
REQUESTS = Counter(
    "scrape_requests_total",
    "Total HTTP requests issued by the crawler",
    ["host", "status"],          # low-cardinality labels only
)
LATENCY = Histogram(
    "scrape_request_duration_seconds",
    "Fetch latency in seconds",
    ["host"],
    buckets=(0.1, 0.25, 0.5, 1, 2.5, 5, 10),
)
IN_FLIGHT = Gauge(
    "scrape_in_flight_requests",
    "Requests currently awaiting a response",
)

def status_class(code: int) -> str:
    # Collapse to 2xx/4xx/5xx-style buckets, but keep 429 explicit
    return "429" if code == 429 else f"{code // 100}xx"

def fetch(client: httpx.Client, url: str) -> httpx.Response | None:
    host = urlsplit(url).hostname or "unknown"
    IN_FLIGHT.inc()
    start = time.perf_counter()
    try:
        resp = client.get(url, timeout=10)
        LATENCY.labels(host=host).observe(time.perf_counter() - start)
        REQUESTS.labels(host=host, status=status_class(resp.status_code)).inc()
        return resp
    except httpx.RequestError:
        REQUESTS.labels(host=host, status="error").inc()
        return None
    finally:
        IN_FLIGHT.dec()

if __name__ == "__main__":
    # Serve /metrics on :8000 for Prometheus to scrape.
    # Bind to loopback or an internal interface — never expose publicly.
    start_http_server(8000, addr="127.0.0.1")
    with httpx.Client(headers={"User-Agent": "compliant-crawler/1.0 (+https://example.com/bot)"}) as client:
        for url in ["https://httpbin.org/status/200", "https://httpbin.org/status/429"]:
            fetch(client, url)
            time.sleep(random.uniform(1, 2))  # polite pacing between requests
        time.sleep(3600)  # keep the metrics endpoint alive for scraping

Point a Prometheus scrape job at 127.0.0.1:8000 and the series above start flowing. The Gauge also works as a decorator or context manager — with IN_FLIGHT.track_inprogress(): — if you prefer that idiom.

Verification & Testing #

The fastest check is to curl the endpoint and confirm your metric families render in the text exposition format:

Checks before wiring an alert to a metricInconsistent labels between workers are the usual cause of a broken panel.Checks before wiring an alert to a metric✓The endpoint parses cleanly with a metrics linter✓Counter values only ever increase within a process life✓Labels are identical across every worker in the fleet✓A restart does not produce a false spike in rate queries
Inconsistent labels between workers are the usual cause of a broken panel.
curl -s http://127.0.0.1:8000/metrics | grep '^scrape_'

You should see scrape_requests_total, the histogram’s _bucket/_sum/_count families, and scrape_in_flight_requests. For a unit test, prometheus_client exposes helpers so you can assert a counter moved without standing up an HTTP server:

from prometheus_client import REGISTRY

def test_request_counter_increments():
    before = REGISTRY.get_sample_value(
        "scrape_requests_total", {"host": "httpbin.org", "status": "2xx"}
    ) or 0
    fetch(client, "https://httpbin.org/status/200")
    after = REGISTRY.get_sample_value(
        "scrape_requests_total", {"host": "httpbin.org", "status": "2xx"}
    )
    assert after == before + 1

To confirm the histogram is usable for percentiles, query histogram_quantile(0.95, rate(scrape_request_duration_seconds_bucket[5m])) in Prometheus and check it returns a bounded number rather than NaN.

Compliance & Operational Guardrails #

  • Bind the metrics server to loopback or an internal network interface. /metrics can leak target hostnames and traffic volumes, so treat it as internal telemetry, not a public endpoint.
  • Keep every label free of personal data. Because a label value spawns a permanent time series, putting a scraped email, username, or full URL into one both breaches data-minimization expectations and creates an unbounded-cardinality outage risk.
  • Emit a 429-class counter so retry behavior stays observable, and feed it into your exponential backoff and retry logic so throttling decisions are driven by measured data rather than guesswork.

Multiprocess Crawlers Need the Multiprocess Collector #

The default registry lives in one process’s memory. A crawler that forks workers — a Celery pool, a Gunicorn-style prefork model, a multiprocessing pool — therefore exposes whichever worker happened to serve the scrape, and the numbers jump erratically between scrapes as different children answer. The symptom is a counter that appears to decrease, which is impossible for a counter and immediately confusing.

The fix is the multiprocess collector, which aggregates per-process files from a shared directory.

import os
from prometheus_client import CollectorRegistry, multiprocess, generate_latest, Counter, Gauge

# Must be set BEFORE any metric is created, in every process.
os.environ.setdefault("PROMETHEUS_MULTIPROC_DIR", "/run/crawler/metrics")

REQUESTS = Counter("crawl_requests_total", "Crawl requests", ["host", "status_class"])
QUEUE_DEPTH = Gauge("crawl_queue_depth", "URLs waiting per host", ["host"],
                    multiprocess_mode="livesum")     # sum across LIVE children only

def metrics_payload() -> bytes:
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return generate_latest(registry)

def on_worker_exit(pid: int) -> None:
    """Remove a dead worker's files so its gauges stop being counted."""
    multiprocess.mark_process_dead(pid, os.environ["PROMETHEUS_MULTIPROC_DIR"])

Two details cause most of the trouble. The directory must be set before the first metric object is constructed — a metric created at import time under the default registry will not participate, and the failure is silent. And gauges need an explicit multiprocess_mode: livesum for anything additive like queue depth, max or min for high-water marks, liveall when each worker’s value is separately meaningful. Leaving it at the default produces a gauge whose meaning changes with the number of workers.

Clean the directory on start-up and call mark_process_dead when a worker exits, or files from dead workers keep contributing to gauge aggregates indefinitely — a crawler that restarts its pool hourly will show a queue depth that grows all day without a single URL being queued.

Short-Lived Jobs and the Push Gateway #

A batch crawl that runs for four minutes and exits will never be scraped. The push gateway exists for this case, but it holds whatever was last pushed forever, so a job that stops running leaves stale metrics that look like a healthy pipeline. Set a grouping key that includes the job name and instance, delete the group when the job completes successfully, and alert on the age of the pushed metrics rather than their values. Where the job runs often enough, a far simpler option is to keep the process alive for one extra scrape interval after the work finishes and let the normal pull path collect it.

Common Mistakes #

  1. Labeling by URL or record identifier. This is the classic cardinality bomb — millions of one-off series that bloat memory and slow every query. Label by host and status class instead.
  2. Creating metric objects inside the loop. Re-registering the same name raises Duplicated timeseries in CollectorRegistry; define metrics once at module scope.
  3. Using a Gauge for something that only increases. Totals belong in a Counter so rate() and reset detection work; a gauge silently loses the reset semantics Prometheus relies on.

Frequently Asked Questions #

How do I expose metrics from a multiprocess or Gunicorn-based scraper? #

Set the PROMETHEUS_MULTIPROC_DIR environment variable to a writable directory and use prometheus_client.multiprocess.MultiProcessCollector. Each worker writes to shared mmap files that a single collector aggregates, so counters are not undercounted across forks. Without this, every worker reports only its own slice.

What histogram buckets should I choose for fetch latency? #

Buckets must bracket the latencies you actually care about. For web fetches, a spread like 0.1, 0.25, 0.5, 1, 2.5, 5, 10 seconds captures fast responses through slow-timeout territory. Pick edges near your alerting thresholds, since histogram_quantile interpolates within a bucket and coarse edges blur the percentile.

Counter, Histogram, or Summary for latency? #

Prefer Histogram. A Summary computes quantiles inside each process and those quantiles cannot be aggregated across workers, whereas a Histogram ships raw buckets that Prometheus percentiles server-side — the only correct approach for a distributed crawl fleet.