Skip to main content

ADR-0154: BrightData SERP dedicated concurrency lane + timeout alignment

Date: 2026-07-28 Status: Accepted Deciders: Adrian (Soft4U), Claude (Opus 4.8)

Decision context:

  • Latency: the primary target — removes a ~123s average SERP queue-wait and ~22 self-inflicted per-run SERP timeouts (each ~25–50s of wasted wall-clock). No new p50/p95 cost added (fewer retries, wider lane).
  • Dependency surface: none new. Uses the existing asyncio.Semaphore + settings; adds one serp_slot() context manager alongside the existing brightdata_slot().
  • Debuggability: SERP queue-waits and slot acquisitions now log under a distinct label ("BrightData SERP slot/queue-wait …"), separated from the scraping gate.
  • Reversibility: single config flip — set brightdata_serp_max_concurrent=3 (or BRIGHTDATA_SERP_MAX_CONCURRENT=3) to restore near-prior behaviour; timeouts revert via brightdata_serp_timeout_seconds/_read_timeout_seconds. No migration.
  • Blast radius: additive. Only adverse_media_agent._brightdata_serp_search moves from brightdata_slot to serp_slot; the scraping callers (social/person/enrichment) are byte-unchanged. Recall logic (queries, escalators, two-lane attribution, never-suppress fallbacks) untouched.
  • Alternative considered: raise the shared semaphore's [1,3] cap (rejected — it would over-parallelise genuinely heavy MCP scraping, the failure the cap was built to prevent).

Context​

Investigations were perceived as "BrightData rate-limited". Live evidence from a funded account contradicts that diagnosis:

  • Account funded ($26+ balance, active), so the SERP API ceiling is 100 QPS (docs: unfunded = 1,000 req/min, funded = 100 QPS). A whole investigation makes a few dozen SERP calls over minutes — nowhere near the ceiling.
  • Worker logs over an OB Holding run: ~0 real HTTP 429s, 0 auth/403 errors (the expired "Agents" free-trial is not in the path), and 22 BrightData SERP TimeoutError events — of which 0 exhausted their retry budget (every one succeeded on retry).

So the bottleneck is per-request latency, not rate rejection. Three compounding defects produced it:

  1. Timeout inversion. adverse_media_agent set the MCP streamable-HTTP read_timeout=40s but wrapped the call in asyncio.wait_for(..., 25s). The outer ceiling sat below the layer that governs the call, so a slow-but- succeeding SERP response (25–40s under concurrent load) was aborted and retried — pure wasted wall-clock, and a risk of dropping a native-language enforcement hit if all retries were consumed on a slow-but-fine call.
  2. Conflated concurrency lane. A single process-global Semaphore(3) (ADR-0077/0095) gated BOTH heavy MCP browser scraping (social-intelligence, person-validation, enrichment) AND the lightweight SERP search_engine HTTP burst (~14 calls). The SERP burst queued behind the 3 scraping slots and ran nearly serially (~123s average queue-wait, measured 2026-06-29), and each slow call blocked the next.
  3. A ceiling built on a misdiagnosis. The Semaphore was hard-clamped to [1,3] because a 2026-04-21 4-case batch failed at 4 concurrent. That batch failed with timeouts (latency), not 429s — the same latency root cause, mis-attributed to account-quota exhaustion.

Decision​

Give SERP its own concurrency lane, separate from MCP scraping, and align the timeouts, all config-driven:

  1. Dedicated SERP lane. New serp_slot() in app/services/brightdata_concurrency.py, backed by a distinct semaphore sized by BRIGHTDATA_SERP_MAX_CONCURRENT (default 6, clamp [1, 32]). adverse_media_agent._brightdata_serp_search acquires serp_slot; the heavy scraping agents keep the tight brightdata_slot ([1,3], unchanged). SERP lives within the 100 QPS budget, so a wider lane is safe; the [1,32] clamp still bounds a misconfig.
  2. Timeout alignment. The outer asyncio.wait_for ceiling (brightdata_serp_timeout_seconds, default 45s) is held >= the MCP read_timeout (brightdata_serp_read_timeout_seconds, default 40s) — the MCP layer governs the call; we never abort one it would have completed. The agent enforces the invariant with max(timeout, read_timeout + 5) — a 5-second headroom above the read timeout, because the outer wait_for measures the whole direct_call_tool operation (connection/session setup + potentially several reads) while read_timeout is only HTTPX's per-read ceiling; max() alone would let an operator collapse them to equal and re-introduce the self-inflicted-timeout bug. The retry budget is bounded [1, 10].
  3. Config-driven knobs. brightdata_serp_max_concurrent, brightdata_serp_read_timeout_seconds, brightdata_serp_timeout_seconds, brightdata_serp_retries in config.py, tunable without code changes.

Recall behaviour is unchanged: same queries, same escalators, same ADR-0078 two-lane attribution, same fail-closed data-gap fallbacks (a sustained outage still fails closed to a data gap — never a false clear).

Consequences​

Positive​

  • Removes the SERP queue-wait (own lane) and the self-inflicted timeout/retry cycle (aligned timeouts) — the dominant avoidable latency in the adverse-media phase (~48% of investigation wall-clock, 2026-06-29).
  • Correctly separates two workloads with opposite profiles; each is sized to its real constraint (SERP → 100 QPS budget; scraping → BrightData scraping tolerance).
  • Fully config-tunable and reversible; distinct log labels aid future measurement.

Negative​

  • A wider SERP lane means more simultaneous SERP calls to BrightData; if their SERP backend contends under high concurrency, per-call latency could rise. The default (6) is deliberately modest, not the full 100 QPS budget; re-measure before raising BRIGHTDATA_SERP_MAX_CONCURRENT.
  • Two semaphores instead of one — a small amount of additional module state.

Neutral​

  • The brightdata_slot scraping gate and its [1,3] clamp are unchanged; this ADR does not revisit the scraping tolerance, only the SERP lane.

Alternatives Considered​

Alternative 1: Raise the single shared Semaphore cap above 3​

  • Rejected: it would over-parallelise genuinely heavy MCP browser scraping (LinkedIn/Crunchbase), which is the workload the [1,3] cap was built to protect. SERP and scraping have opposite tolerances; a shared knob can't serve both.

Alternative 2: Just raise _SERP_TIMEOUT and keep the shared lane​

  • Rejected: it fixes the timeout inversion but not the ~123s queue-wait — the SERP burst would still serialise behind the scraping gate.

Alternative 3: Switch SERP to BrightData async request mode​

  • Deferred: async (submit + poll a designated endpoint) genuinely frees the connection during a slow fetch, but it is a larger reworking of the call site than the latency problem warrants now. Revisit if the dedicated lane + timeout alignment prove insufficient under multi-case load.