Skip to main content

ADR-0095: Investigation performance & scale hardening

Date: 2026-07-06 Status: Accepted Deciders: Adrian (Soft4U), Claude Opus 4.8

Context​

A live OB Holding investigation showed ~15–20 min wall-clock with a 3-minute dead zone in which ~6 agents queued ~125s each for a BrightData slot. A code-research pass (4 agents + synthesis, verified against the source) established the scale behaviour precisely:

  • Per-investigation latency degrades LINEARLY under concurrent load, not exponentially — every fan-out is hard-capped (network depth ≤2, Tier-B ≤8 targets, retries ≤3), so there is no N² explosion inside a case.
  • The prime bottleneck is a process-global asyncio.Semaphore(3) that serializes ALL BrightData work across the whole worker (brightdata_concurrency.py:64, clamped to [1,3] — the real plan quota). One investigation alone fires ~5–6 simultaneous requests against those 3 slots. NorthData adds a second process-global lock (30 req/min).
  • The one genuine exponential vector is retry amplification: run_osint_investigation ran with a 30-min timeout, maximum_attempts=3, and NO mid-flight checkpoint — so a contention-induced timeout re-runs the ENTIRE external burst, re-injecting BrightData demand and pushing other cases past THEIR timeout (a cascade that can start at N=2).
  • The single-machine target is a MacBook Air M4 / 32 GB; Docker Desktop is allocated 15.6 GB and the 11 infra containers had no memory limits (one could OOM the rest; ClickHouse alone used 2.23 GB uncapped). The app tier runs locally (not in Docker).

Decision​

Harden performance at every level as five workstreams:

  • W1 — Reliability (defuse the exponential vector). A dedicated RetryPolicy(maximum_attempts=1) for run_osint_investigation (the activity already fails closed internally, so a whole-activity retry is redundant + the cascade fuse), plus an admission-control semaphore (settings.osint_max_concurrent_investigations=2) so no more OSINT bursts hit the BrightData gate than can finish under the 30-min timeout.
  • W2 — Sizing for M4/32 GB. Per-container mem_limit/mem_reservation on all 11 infra services (12 GB total, 3.6 GB headroom in the 15.6 GB Docker budget); Postgres memory tuning (shared_buffers=512MB, effective_cache_size, work_mem, SSD effective_io_concurrency); Neo4j heap 1 GB / page-cache 512 MB; ClickHouse capped (cgroup-aware) from 2.23 GB → 1.5 GB; Redis configured as a cache tier (maxmemory 512mb, volatile-lru).
  • W3 — Cross-investigation caching. Redis-cache NorthData registry lookups by (reg-id|name|country) (external_cache.cached_external, 24 h TTL, fail-open) to cut the 30-req/min NorthData gate and let same-group cases reuse scans. Compliance boundary: only factual/slow-changing data is cached cross-investigation; the helper supports scope=case_id for within-investigation-only caches so a stale "no adverse media" can never suppress a fresh risk signal in a later case.
  • W4 — Parallelize Tier-B recall. The per-target loop runs concurrently under a bounded semaphore; BrightData stays capped by its own gate, but the Tavily + LLM-analysis portions overlap. gather preserves order → deterministic attribution.
  • W5 — Cross-process rate limits (documented, deferred). The BrightData semaphore and NorthData lock are per-process. Horizontal scale-out (K workers) would multiply — not share — the real account quota, causing 429s that trip breakers into silent data-gaps. A Redis token-bucket is the prerequisite for safe multi-worker scaling. Deferred: the current target is a single machine (one heavy worker; the backend does little BrightData), so per-process limits are effectively the account limits today.

Decision context:

  • Latency: W1/W4 cut per-case wall-clock and eliminate the cascade; W3 cuts the concurrent-load curve. Estimated steady-state ~7 investigations/hour/worker, degrading gracefully.
  • Dependency surface: no new packages; reuses the existing cache_service (Redis), circuit breakers, and Temporal retry policies. One new helper module + config knobs.
  • Debuggability: admission logs slot-waits; the cache is fail-open; container limits make OOM attributable to one service instead of a shared crash.
  • Reversibility: every lever is a config knob or a mem_limit; caching has an external_cache_enabled kill-switch.
  • Blast radius: W1 changes retry behaviour (a hard OSINT failure no longer retries — acceptable vs. the cascade); W2 recreated the infra containers (verified healthy). Additive otherwise.
  • Alternative considered: raise the BrightData Semaphore(3) clamp — rejected (it is the real plan quota; the code comment records that 4 broke a batch). Caching, not a higher clamp, is the demand-side lever.

Consequences​

Positive​

  • The exponential cascade vector is eliminated (W1) — latency now degrades predictably and linearly.
  • Container memory is bounded and attributable; ClickHouse no longer free-runs to 2.23 GB.
  • NorthData load drops for same-group / repeat investigations; Tier-B recall is faster.
  • The system is honestly sized for the M4/32 GB it actually runs on.

Negative​

  • W1: a genuine hard OSINT failure (e.g. worker crash mid-activity) no longer retries — the workflow fails and the officer re-runs. Accepted as far better than a system-wide cascade under load.
  • W3: a 24 h NorthData cache can serve registry data up to a day stale; mitigated by the moderate TTL, the empty-result guard, and monitoring re-runs after TTL. Adverse-media is deliberately NOT cross-investigation-cached.
  • W5 is deferred — horizontal scale-out is not yet safe without the cross-process limiter.

Neutral​

  • Worker activity concurrency stays adaptive (4 on ≥24 GB); the OSINT admission semaphore is now the external-API limiter, decoupling memory-concurrency from BrightData-concurrency.

Alternatives Considered​

Alternative 1: Raise BrightData/worker concurrency to "go faster"​

  • Rejected: the Semaphore(3) is the real BrightData plan quota (comment: 4 broke a batch); raising worker concurrency without it just converts in-process queue-wait into upstream 429s and breaker-tripped data-gaps. The levers are caching (demand) and admission control (safety), not a higher ceiling.

Alternative 2: Checkpoint-resumable OSINT activity (skip completed agents on retry)​

  • The ideal long-term fix for retry cost. Deferred as a larger change; W1 (maximum_attempts=1) removes the acute cascade risk now for near-zero effort.