Mixed accuracy; large token savings against a constrained page-fetch agent. Later tests found no accuracy advantage over full reading or simple RAG.
Early tool-use experiment
The baseline had a docs root and URL fetch, with no search engine, browser, local index, or context compaction. Scull supplied compact lookups. These measure those interfaces, not a capable browsing agent. Domains are anonymized; source-specific reports and transcripts remain excluded.
| Domain | Task pass rate: Scull / page fetch | Agent tokens: Scull / page fetch |
|---|---|---|
| Analytical engine | 96.2% / 78.9% | 1.9M / 41.4M |
| Developer CLI | 87.4% / 87.4% | 2.2M / 44.6M |
| Niche language | 81.2% / 95.0% | 13.9M / 31.2M |
The language artifact needed content fixes before the retained sweep. Compact facts did not consistently help agents produce working code.
The September 5–6, 2026 ledger contains 304, 95, and 240 runs per condition respectively: 639 per condition, 1,278 total. The first domain uses equal-weighted model averages; the others pool passes. Do not sum the displayed metrics across domains. Method.
Later full-read and RAG tests
Representative same-model results from separate decision-focused suites:
| Anonymized corpus | Full read | RAG | Scull | Agent tokens: full read / RAG / Scull |
|---|---|---|---|---|
| Reviewed API docs | 12/12 | 12/12 | 12/12 | 211,397 / 95,940 / 69,145 |
| Synthetic adversarial docs | 10/10 | 10/10 | 10/10 | 15,261 / 16,407 / 40,761 |
No accuracy lift was observed across the broader four-model evaluation program; not every model ran in every condition. A later size sweep found Scull did not beat RAG on the tested corpus. Extra tool turns can erase smaller-result savings. These findings led development to stop.
What remains worth testing
Compact local lookup may suit agents without web access or with small context budgets. The existing results do not establish an advantage for private docs, local models, or air gaps. A next test should compare the same local model and pinned docs through Scull and reasonable local retrieval, including build/review effort and actual disconnected operation.