# Changelog — legal-analysis-forge

All notable changes to this skill are documented here.

Format: `## [vX.Y] — YYYY-MM-DD`

---

## [v1.1] — 2026-05-22

Polish release. Fixes the one real skill bug surfaced by the v1.0 iteration-1 sweep: the plain-English explainer routinely overshoots its own 150–300 word target.

**What changed:**

- **SKILL.md Step 5 (Execute)**: the explainer length rule moved from a permissive *"150–300 words (default; adjust to complexity of the analysis)"* to a hard upper bound. The previous phrasing gave the executor an escape clause (*"adjust to complexity"*) that observed overshoots (486 / 500 / 597 / 610 words across iter-1 evals 0/1/2/3) exploited consistently. New phrasing: *"150–300 words. This is a hard upper bound, not a soft target. If you find yourself exceeding 300 words, cut — do not negotiate with yourself that 'this analysis was complex enough to justify more'. A 300-word ceiling forces the explainer to do the one job it is for: tell the practitioner, in one screenful of plain language, what the document is and what the analysis found. Anything longer is the formal analysis duplicating itself. Count words before saving; if over 300, revise down."*
- **`references/analytical_canon.md` self-check item 12**: extended to explicitly enforce the 150–300 word bound (*"count before saving; revise down if over 300 — the cap is a hard upper bound, not a soft target"*). Previously the self-check checked only that the explainer existed.

**Why this matters:** The explainer's job is to give the practitioner a one-screenful orientation alongside the formal deliverable. When it doubles in length, it duplicates the analysis it accompanies — defeating the design intent. Forcing the count discipline at both the instruction site and the self-check site catches the overshoot before save.

**What did not change:** the substantive deliverable quality (formal analysis, prompt structure, citation conventions, register, anti-hallucination protocol, bilingual handling) — all unaffected. The v1.0 design stands; v1.1 is targeted polish.

**Iteration-1 eval-set issues parked for a separate iter-2 commit (not part of v1.1):** eval-3 doctrinally contestable legal-bases assertion + missing Art. 26 deployer-duties assertion; eval-8 routing/content assertion tension (split into routing-only + explicit-override variants).

### Iter-2 sweep result (2026-05-22)

Iter-2 benchmark sweep run against the cleaned-up 11-eval / 110-assertion set, with-skill (v1.1) vs no-skill baseline.

| Metric | With skill (v1.1) | No-skill baseline | Δ |
|--------|--------------------|--------------------|---|
| Pass rate (mean of 11 evals × 110 assertions) | **92.2% ± 9%** | 53.8% ± 28% | **+38.4pp** |
| Wall-clock per case | 234.3s ± 86.4s | 129.6s ± 64.8s | +104.7s |
| Output chars per case (token proxy) | 31,504 ± 17,247 | 16,271 ± 8,348 | +15,232 |

Vs iter-1: with-skill mean lifted **+4.9pp** (87.3 → 92.2); differential **widened +2.9pp** (35.5 → 38.4); with-skill stddev tightened from 13% → 9%; cost ratio dropped from 2.47× wall-clock to 1.81×.

**v1.1 explainer-length fix verified end-to-end.** Every with-skill run that produced an explainer landed inside 150–300 words (range 220–295). Zero overshoots vs iter-1's 486 / 500 / 597 / 610. The two non-producers are correct behaviour: eval-8 routed to `dpa-art28` (no work done, no explainer needed); eval-5 embedded the explainer content inside the characterisation file when Step 5 was skipped (grader accepted this as satisfying the "alongside the prompt" requirement).

**Iter-2 eval-design fixes worked as intended:**

- **Eval-8 routing-only** is now the cleanest discriminator in the set: 6/6 (100%) with-skill vs 0/6 (0%) baseline. Removing the four content-quality assertions that were in tension with the routing rule gave the routing behaviour a clean test surface.
- **Eval-3 closed up from iter-1's +0pp tie to +36pp** — but the predicted Art. 26 deployer-duties discriminator did NOT materialise (both configs still missed Art. 26). The differential came instead from the legal-bases nuance, word-budget compliance, and explainer presence. The Art. 26 coverage gap is a real SKILL.md improvement candidate for a future minor.
- **New eval-10 (explicit-override)** landed at +20pp (10/10 vs 8/10). Both configs proceeded with the analysis since the user spelled out the override; with-skill won on the explainer + citation-verification axis. The companion pair (eval-8 route-when-you-should + eval-10 honour-overrides-when-explicit) now works as a unit.

**Two findings worth flagging for a future minor:**

- **Eval-1 with-skill scored 7/10** (lowest in the with-skill column). The executed self-check applied a 9-point protocol rather than `analytical_canon.md`'s documented 12-point protocol — a SKILL.md instruction-drift signal worth investigating. Also failed: ratio/obiter terminology in the prompt block, and the sibling-routing suggestion (which may be over-strict for purely analytical CJEU memos).
- **Eval-7 design tension**: with-skill executed Step 5 against the assertion's "skip Step 5" expectation. The German "Bitte den Prompt erstellen — ich brauche [list of artefacts]" is genuinely ambiguous; recommendation is to split into eval-7a (prompt-only) and eval-7b (prompt + execute) in a future eval-set revision.

**Workspace**: `skills/legal-analysis-forge-workspace/iteration-2/` (gitignored). 22 runner subagents + 11 grader subagents, ~1.0M tokens total runner cost. Six grader runs wrote `output_chars: null` and were patched with actual output character counts before aggregation; otherwise no incidents. No new tag (per `CLAUDE.md` release ritual — v1.1 is the standing release; iter-2 is the validation pass).

---

## [v1.0] — 2026-05-22

First reviewed release. Promoted from v0.9 (pre-review) after a `/skill-creator` eval-iteration sweep cleared the `project_skill_review_initiative` +10pp gate by a 25.5pp margin.

**Benchmark headline (iteration-1):**

| Metric | With skill | No-skill baseline | Δ |
|--------|------------|--------------------|---|
| Pass rate (mean of 10 evals × 103 assertions) | 87.3% ± 13% | 51.8% ± 17% | **+35.5pp** |
| Wall-clock per case | 258.0s ± 125.4s | 104.4s ± 37.3s | +153.6s |
| Tokens per case | 47,835 ± 17,619 | 21,654 ± 9,771 | +26,181 |

**Where the skill earned its keep:**

- **Canonical use cases cleared by +50pp or more.** Five of the ten evals (consultation response on a draft, internal CJEU memo, DE LinkedIn post on an EDPB guideline, external client memo on a national DPA decision, horizon-scan entry on a draft delegated act) show +50pp or larger differentials. These are the deliverable types the skill was designed for, and the prompt-engineering scaffolding clearly earns its keep on them.
- **Automatic freshness check works in production.** Eval-2: the user referenced "EDPB-Leitlinien 02/2024" — the actual document is Guidelines 1/2024. With-skill caught the numbering error via live web verification and produced the post against the right document; baseline propagated the user's stale framing. Eval-5: with-skill explicitly invoked the freshness protocol on a March-2024 draft and surfaced the version-choice to the user before proceeding.
- **Citation hygiene under the anti-hallucination protocol.** Across eval-1, eval-6, eval-9, the with-skill runs consistently flagged case-law ECLIs as recalled-from-memory and routed them for live verification at curia.europa.eu before any external use. Baselines asserted the same citations as fact. For legal deliverables this is the difference between a memo that holds up under counsel review and one that doesn't.
- **Register discipline holds.** The dry-practitioner register codified in `references/analytical_canon.md` (no em-dashes, no marketing fillers, no banned German phrases like *Selbstverständlich* or *Im Ergebnis lässt sich festhalten*) survives the round-trip. Eval-2 baseline used em-dashes twice as rhetorical device, which the skill register explicitly bans — clean discriminator between the two configurations.
- **Multi-artefact deliverables.** With-skill produces five artefacts per executed run (characterisation, prompt, executed analysis, plain-English explainer, transcript) versus the baseline's single document. For a working DPO or compliance lawyer, that's the difference between "a document I drafted" and "a piece of work I can defend in a board meeting".
- **Output consistency.** With-skill stddev is 13% vs baseline 17%. Narrower output variance, same pattern observed with `ai-act-high-risk` v1.0 — predictable structure is itself a value-add for downstream consumers.

**Two anomalies, both eval-design issues rather than skill defects:**

- **Eval-3 (comparative analysis Art. 22 GDPR × Art. 50 AI Act) tied at 70/70.** Grader flagged the assertion on differential legal bases (Art. 16 vs Art. 114 TFEU) as doctrinally contestable. Both configurations also failed to cite Art. 26 AI Act deployer duties despite covering Art. 50 deployer transparency extensively. Eval design to sharpen for iteration-2.
- **Eval-8 (routing test) at 60/30.** 6 of 10 assertions test routing-decision behaviour (with-skill nailed 6/6); 4 of 10 assume the skill proceeds to deliver the analysis it just routed away — mutually exclusive under the current SKILL.md. Grader recommended splitting into routing-only + explicit-override variants. On the well-formed subset of 6 assertions, with-skill scored 100%.

**One real skill bug surfaced (parked for v1.1):**

- **Plain-English explainer overshoots length target.** Documented ceiling is 150-300 words; observed lengths across with-skill runs were ~486 (eval-3), ~500 (eval-1), ~597 (eval-2), ~610 (eval-0). The explainer is produced reliably but doesn't honour the upper bound. Worth fixing for v1.1; does not block v1.0 promotion since substantive deliverable quality is unaffected.

**No skill content changes for v1.0** — the benchmark validated the v0.9 design without surfacing changes needed to clear the gate. The two eval-design issues will be addressed before any iteration-2 benchmark; the explainer-length bug will be fixed in v1.1.

**For context vs sibling promotions:**

| Skill | v1.0 promotion differential | Notes |
|-------|------------------------------|-------|
| `breach-sentinel` | various iterations to v2.2 | longest-running review subject |
| `ai-act-high-risk` | +20.6pp | depth-specialist on Art. 6 |
| `legal-analysis-forge` | **+35.5pp** | first meta-skill in the portfolio |

---

## [v0.9] — 2026-05-21

Initial pre-review release. First skill in the portfolio whose deliverable is a **tailored expert prompt** for structured legal analysis of an EU digital regulation document, rather than a direct operational compliance artefact.

- **Scope**: EU digital regulation only — GDPR, AI Act, Data Act, DGA, DSA, DMA, NIS2 (incl. BSIG-neu), ePrivacy, CRA, DORA, eIDAS 2.0, PLD (Dir. 2024/2853), AI Liability Directive, and adjacent secondary instruments (delegated acts, implementing acts, harmonised standards under Art. 40 AI Act, codes of conduct under Art. 40 GDPR / Art. 56 AI Act / Art. 45 DSA). Out of scope: competition, IP, tax, employment, sectoral law not touching the digital stack.
- **6-step workflow**: Ingest → Characterise → Elicit (≤3 questions) → Generate prompt → optional Execute → optional Refine. Steps 5 and 6 are skipped when not needed.
- **Three reference files**:
  - `references/document_taxonomy.md` — instrument types (Regulation, Directive, Decision, delegated/implementing acts, Commission/EDPB/AI Office/ENISA guidance, CJEU/AG/national court judgments, DPA decisions, codes of conduct, harmonised standards, draft/consultation instruments, Member State transposition), binding force, characterisation signals, scrutiny points.
  - `references/outcome_templates.md` — prompt skeletons for 10 outcome types (stakeholder consultation response, internal compliance memo, external client memo, public commentary, conference talk prep, internal risk assessment, litigation brief input, comparative analysis, horizon-scan entry, skill input).
  - `references/analytical_canon.md` — interpretive canon (hierarchy, text vs interpretation, legal basis, temporal application, cross-instrument coherence, language versions, gaps, operability, authority weighting, drafting history, carve-outs), register constraints (prohibited and required vocabulary/structure, EN and DE), citation conventions, 12-point self-check protocol, bilingual handling (EN/DE terminology maps), anti-hallucination protocol.
- **Automatic freshness check** for draft and consultation documents in Step 1 — surfaces later finalised versions from authoritative sources before analysis proceeds.
- **Opt-in live research** on authoritative sources (eur-lex.europa.eu, curia.europa.eu, edpb.europa.eu, ec.europa.eu sub-domains, ENISA, BEREC, national DPAs, national gazettes) to avoid reliance on stale training data; commercial trackers and academic journals permitted for context only; Wikipedia, LinkedIn, news, and aggregator blogs excluded.
- **Plain-English explainer** produced alongside every executed analysis; integration into the deliverable (executive summary / sidebar / annex / cover note / standalone) is the user's choice.
- **Integration with sibling skills**: surfaces downstream operational tasks (LIA, TIA, DPA review, RoPA update, AI Act classification, NIS2 scope analysis, breach response, Data Act compliance) and suggests the relevant compliance skill *after* the analysis is delivered, not before.
- **Outputs**: written to the user's current working directory by default (`./[doc_slug]_prompt_[outcome_slug].md`, `./[doc_slug]_analysis_[outcome_slug].md`, `./[doc_slug]_plain_english_[outcome_slug].md`); refinements append `_v2`, `_v3` suffixes; stable slugs track a single document through multiple analyses.
- **Evals**: stub set of 3 cases in `evals/evals.json` covering (1) consultation response on a draft instrument with bilingual handling, (2) internal compliance memo on a CJEU judgment, (3) German LinkedIn post on a new EDPB guideline. Full 8–10-case sweep will follow before v1.0 promotion.

Status: pre-review, awaiting eval-iteration sweep against the `project_skill_review_initiative` +10pp gate before v1.0 promotion.