Skip to main content

ADR-0120: Recorded robots.txt exception for ejustice.just.fgov.be

Date: 2026-07-22 Status: Accepted Deciders: Adrian (Soft4U BV) — decision; Claude (Opus 4.8) — analysis Implements: ADR-0119 §7 D1

Context​

The Belgian ITAA rules (professional and deontological requirements for accountants and tax advisers) are cited by reasoning_template_registry as "ITAA Regulations". They are not a private body's PDFs: they are Royal Decrees and an Act published in the Moniteur belge, and the ITAA's own "Législation" page links to them as ejustice ELI URLs rather than hosting copies. Eight instruments were located and verified reachable (HTTP 200 with real deontology text retrieved), including:

InstrumentELI
Wet 17 maart 2019 — the ITAA framework Act/eli/wet/2019/03/17/2019040805/justel
KB 1 maart 1998 — reglement van plichtenleer der accountants/eli/besluit/1998/03/01/1998016071/justel
KB 14 januari 2021 — approving the BIBF plichtenleer/eli/besluit/2021/01/14/2021040128/justel

Alternative channels were investigated and rejected on the evidence:

  • itaa.be renders its rules through a JavaScript flipbook (dFlip); the served HTML contains 21 characters of text and no PDF URL. Its WordPress REST API returns HTTP 401.
  • etaamb.openjustice.be, a third-party mirror with no robots.txt, carries the full text but interleaves promotional blocks into the legal text (…bedoeld in artikel 13 van de wet van 22 juli 1953 [Relevante gevonden documenten / type wet / prom. 22/07/1953 …] houdende oprichting…). Ingesting it verbatim would corrupt article bodies — precisely the class of defect the ADR-0118 gate exists to catch.

The blocker is policy, not capability. https://www.ejustice.just.fgov.be/robots.txt (HTTP 200) disallows, for User-agent: *:

Disallow: /eli/       Disallow: /cgi_loi/     Disallow: /cgi_wet/
Disallow: /cgi/       Disallow: /loi/         Disallow: /wet/
Disallow: /mopdf/

That is every path on which Belgian legislation lives, for every agent — not an AI-specific exclusion.

Decision​

Ingest the ITAA instruments from ejustice as an explicit, recorded exception, subject to the guardrails below. The decision is the product owner's, taken with the trade-off stated; it is recorded here rather than left implicit in a fetcher.

Rationale accepted:

  1. The material is public law from a government publisher — statute, not proprietary content, and Belgian legislative text carries no copyright.
  2. The use is a low-volume internal reference corpus: eight instruments, fetched once. We neither operate a search engine over the site nor republish its text.
  3. Every compliant alternative has been checked and none works (above).

Binding guardrails​

These are requirements on the implementation, not aspirations:

  • Fetch once, cache durably. Ingestion is one-shot per instrument with the result persisted. No recurring crawl, no scheduled re-fetch. Re-ingestion is a manual, deliberate act.
  • Rate limit ≥ 5 s between requests, sequential only. Also an operational necessity: ejustice timed out twice (25 s, 30 s) during research, so the fetcher budgets 60–90 s timeouts with retries.
  • Honest identifying User-Agent with a contact address. No UA spoofing, no pretending to be a browser. If the operator wants to identify or block us, they can.
  • No redistribution. Text stays in the internal corpus and is surfaced to officers as citations. It is not republished, and the case-pack export carries citations rather than reproduced statute.
  • One-flag reversibility — lex_ejustice_enabled (default False). If FOD Justitie objects, the channel is disabled with a config change, not a deployment.
  • Encoding. ejustice serves ISO-8859-1. It must be decoded explicitly; a UTF-8 assumption silently mangles Dutch/French text (and, as this project learned on the German corpus, produces false "content not present" conclusions).

Open action, not a dependency​

Request written permission or the Justel bulk product from FOD Justitie, which would convert this exception into a sanction. Recommended and recorded; it does not block.

What this decision does NOT extend to​

zakonyprolidi.cz remains disqualified and this ADR must not be cited to justify it. It names ClaudeBot specifically with Disallow: /, sets Content-Signal: ai-train=no, use=reference, and asserts a DSM Article 4 reservation of rights. That is an express, targeted refusal, categorically different from a generic crawler convention — and the official e-Sbírka open data supersedes it at no loss (ADR-0119, Wave 4).

The distinction is the point of this ADR: a blanket Disallow: /eli/ on a government publisher with no machine-readable alternative is one thing; a site naming our agent and reserving its rights is another.

Consequences​

Positive​

  • The ITAA citations become groundable, closing the last corpus-coverage gap.
  • The decision, its rationale and its guardrails are auditable — which is what EU AI Act Art. 12 asks of a system whose outputs must be traceable.
  • The flag makes the position reversible in minutes.

Negative​

  • We act against a stated site policy. That is a real cost and is not neutralised by the material being public; it is accepted deliberately, with a named decision-maker and a date.
  • The KNOWN_UNGRAMMATICAL register loses an entry, so the citation contract no longer surfaces this as an open question — the ADR becomes the only record.
  • ejustice is slow and intermittently unavailable; ingestion will occasionally fail and need re-running.

Neutral​

  • No change to any rule, severity or risk computation. This governs where one instrument's text comes from.

Alternatives Considered​

Alternative 1: Skip ITAA, record it as an honest gap​

Leave the citation unverifiable and say so. Rejected by the product owner, but it was the recommended option and remains the fallback if the flag is turned off. Cost: one instrument-level citation stays ungrounded.

Alternative 2: One-off manual archival​

A human fetches the eight instruments in a browser; we ingest from local files. No automated request is made, so no robots question arises. Rejected as the primary route because the corpus then goes stale silently unless someone repeats the exercise — the corpus-staleness failure mode this project has already been bitten by.

Alternative 3: Use the etaamb mirror​

No robots restriction. Rejected on data quality: it interleaves promotional blocks into legal text, and a corpus that looks populated while carrying corrupted articles is the exact defect class of NL-Wwft and DE-GwG.

Decision context​

  • Latency: eight sequential fetches at ≥ 5 s spacing, once. Not on any request path.
  • Dependency surface: none new; httpx + the existing HTML parser.
  • Debuggability: a disabled flag raises a named error stating the ADR, so an operator who finds ITAA missing learns why immediately.
  • Reversibility: minutes — one config flag.
  • Blast radius: one instrument. No other corpus or code path is touched.
  • Alternative considered: skip and record as a gap (Alternative 1) — rejected by the product owner in favour of coverage, with guardrails.

References​

  • ADR-0119 §7 D1 (the decision this implements)
  • ADR-0118 (corpus integrity gate — why the etaamb mirror is disqualified)
  • Issue #484