Design Output — Flight 1: KB Document Format

Purpose. Should a knowledge-base document (injected in-context) be formatted as long-form prose, structured Markdown, or semantic XML? This decision informs the format used for product/PRD-style reference documents fed to an AI assistant answering questions against them.

What's under test. Three arms, one variable (document_format), same underlying facts (a fictional product's PRD — "OnboardIQ," an AI-powered employee onboarding tool):

Arm A (Control) — Long-form Prose Markdown
Arm B (Treatment 1) — Structured Markdown
Arm C (Treatment 2) — Semantic XML

Models tested. Claude Sonnet 5 and Claude Haiku 4.5 (full cross with all three arms).

Content/task set. A single PRD document (OnboardIQ) rendered in all three formats, tested against 7 questions spanning simple lookup, audience-tailored synthesis, cross-section synthesis, precise ID extraction, numeric retrieval, a faithfulness/hallucination trap, and open reasoning.

ID	Type	Description
Q1	simple_lookup	Security/compliance requirements
Q2	audience_synthesis	Launch prerequisites, tailored for a Customer Success audience
Q3	cross_section_synthesis	Top 3 risks and their KPI linkages
Q4	precise_extraction	P0 requirement IDs
Q5	numeric_retrieval	Time-to-productivity KPI target vs. baseline
Q6	faithfulness_check	Whether a mobile app is specified (it is not — a hallucination trap)
Q7	open_reasoning	Which persona is most underserved, and why

Quality rubric. The 5-criterion blind rubric from evaluation_framework.md (correctness, completeness, groundedness, audience fit, conciseness), no domain-specific substitutions.

Cost formula. Not measured this flight — see the execution path below.

H1: Structured Markdown (B) will reduce document footprint (chars/tokens) versus Control (A) with no meaningful quality loss.
H2: Semantic XML (C) will improve groundedness/correctness versus Control (A) due to explicit structural tagging, at the cost of increased footprint versus Structured Markdown (B).
H3: Quality will not differ meaningfully by model (Sonnet 5 vs. Haiku 4.5) on this task set.

Execution path used. Claude Code's Agent tool (fallback path per execution_playbook.md, §2) — no raw API key was available in this session. This means: no clean input/output token split (only a blended total_tokens figure, dominated by ~17-23k tokens of fixed subagent overhead per call), no real dollar cost (blank throughout — computing one from the blended total would be misleading), and latency includes orchestration time, not raw model inference.

Known scope limitations going in. 1 repetition per cell (not multiple reps) — reps were cut from an original larger design to fit the execution environment's realistic call volume, per execution_playbook.md's documented pattern from this same flight. Findings are therefore directional/exploratory, not statistically significant.

Why this is in the design, not the footnotes. Naming what the execution path would distort — before running it — is what separates a limitation from an excuse written after the numbers came back.

Total rows: 42 (3 arms × 2 models × 7 test cases × 1 rep) — succeeded: 42, failed: 0
Date: 2026-07-11
Environment / execution path: Claude Code, Agent tool
Deviations from original design: Repetitions cut to 1 per cell before the run, to fit the Agent-tool execution path's realistic call volume.