Early constrained-agent runs
Same task prompts and models; different documentation interfaces:
| Condition | Access |
|---|---|
| Scull | Compact ranked results and named lookups over precompiled schemas |
| Page fetch | Documentation root, URL fetch, and the task execution tool |
The baseline had no search engine, browser, indexed retrieval, cleaned reader view, or context compaction. Fetching returned up to 51,200 bytes with a 15-second timeout; content stayed in history. Scull instructions encouraged targeted search, and some domains had concept-scoped retrieval.
These compare tool-and-prompt packages, not equally capable retrieval systems. The loop defaults to 25 turns; individual runs can override it. Executable scorers checked outputs, not prose quality.
Aggregation and cost
The September 5–6, 2026 runs cover 10 models and 1,278 task-condition runs. The analytical-domain summary averages repeats within each model, then weights models equally. Other domains pool passes. One API-error run was excluded rather than scored as a failure. Do not combine domains into one score.
Cost estimates cover agent inference only. They exclude extraction, review, indexing, embeddings, maintenance, and infrastructure. Token savings are not total deployment savings.
Later comparisons
Decision-focused tests added full-read and naive RAG conditions over small reviewed and synthetic corpora. They found no accuracy advantage; a size sweep found conditional cost savings. These tests answer a different question and must not be pooled with the early runs.
Limits
Source domains are anonymized. Raw inputs, prompts, transcripts, and detailed reports are excluded; this aggregate is not an independently reproducible benchmark. Early schemas were unreviewed model extractions; failures drove corrections in one domain before its final sweep.
Results do not establish performance on private docs, local models, or air-gapped deployments. No web access still allows strong local retrieval. Results.