Score Output

By arm (combined across models, n=14 each):

*Latency and blended-token figures from this execution path are not reliable for absolute or cross-model comparison — see Known Limitations.

By arm × model:

By test-case type (combined, n=6 each):

Per-criterion averages (all rows):

Hallucination rate: 0/42 rows flagged (quality_groundedness < 4) — every arm correctly handled the Q6 faithfulness trap, stating the mobile app is not specified rather than inventing one.

Cost: Not measured — this execution path cannot produce a clean input/output split.

• H1 — Structured MD reduces footprint, no quality loss. Supported
  −47% footprint (12,052→6,426 chars) with quality statistically indistinguishable from Control (4.84 vs. 4.73 — well inside the noise band, given the ceiling effect below).
• H2 — Semantic XML improves groundedness/correctness over Control. Not supported
  Groundedness and correctness did not meaningfully differ by arm — all three arms scored a hallucination rate of 0% and near-identical correctness. XML's structural tagging did not produce a measurable accuracy edge in this test.
• H3 — No meaningful quality difference by model. Supported, with a caveat
  Quality was close across models (4.66–4.89), but latency showed a large, consistent gap — Sonnet 5 appeared faster than Haiku 4.5 in every single arm, which is very likely a measurement artifact of this execution path rather than a real finding.

• Sonnet 5 appeared consistently faster than Haiku 4.5 across all three arms (e.g. Arm A: Sonnet 4,751ms vs. Haiku 8,255ms). This is backwards from real relative inference speed and matches a distortion execution_playbook.md explicitly documents for the Agent-tool execution path — orchestration-inflated latency, not raw model inference time. Treat this flight's latency comparisons as non-representative of true model speed.
• Conciseness was the lowest-scoring criterion on average (4.31), consistent with the evaluation framework's own documented failure mode: judges penalizing accurate elaboration as "verbose" even when nothing added was factually wrong. Several judge_rationale fields say so explicitly — e.g. "slightly verbose with added editorializing phrases" on an otherwise fully correct answer.
• Semantic XML (Arm C) reduced footprint far less than Structured Markdown (−15% vs. −47%) despite being a more heavily "structured" format — the tag overhead of XML appears to eat into the compression gains plain structured Markdown gets for free.

• Single repetition per cell. Findings are directional/exploratory, not statistically significant — a real difference between arms could be hiding inside single-rep noise.
• Execution path distorts token and latency signals. Agent-tool overhead (~17-23k tokens/call) dominates the blended total_tokens figure, so token counts above should not be read as content-driven. Latency includes orchestration time and produced a known-backwards Sonnet/Haiku speed comparison.
• No real dollar cost. modeled_cost_usd is blank throughout — this execution path cannot cleanly split input/output tokens, so no defensible cost figure exists for this flight.
• Ceiling effect. 40/42 rows (95%) scored ≥4.5/5 composite quality. On content this easy for the model to handle correctly, quality differences between arms are compressed and hard to trust — this test content was probably too easy to strongly discriminate between formats on quality alone. The footprint finding is more trustworthy than any quality delta.
• Single, non-independent judge. The judge model was not varied independently from the answer models, so self-preference bias was not controlled for and is an unaddressed limitation of this flight.

On this test set, Structured Markdown cut document footprint by nearly half (−47%) versus long-form prose with no measurable quality cost — the clearest, most trustworthy finding of the flight. Semantic XML did not earn its overhead: it compressed less than Structured Markdown (−15% vs. −47%) and showed no groundedness or correctness advantage to justify the extra tagging weight. Quality itself was not very discriminating this round (ceiling effect, 95% of rows ≥4.5/5) — the footprint result, not the quality result, is what this flight actually proved. Latency and token-cost figures from this run should not be trusted for absolute comparisons, given the execution path used.

• Harder content set. Rerun the same three-arm format comparison against longer, messier, more ambiguous source content to break the ceiling effect and get a real read on whether format affects quality, not just footprint.
• Raw API execution path. Rerun with a real API key available to get clean token splits, real modeled cost, and trustworthy latency — directly resolving this flight's largest limitation.
• Independent judge model. Vary the judge model away from the answer models' family to address the self-preference bias gap.
• Multiple reps. Increase to 2-3 reps per cell (execution path permitting) to move findings from directional to statistically defensible.