# Experiment results

> Documentation index: [llms.min.txt](https://caseyrfsmith.github.io/scull/llms.min.txt) · Document manifest: [documents.json](https://caseyrfsmith.github.io/scull/documents.json)

Breadcrumbs: The experiment > Experiment results

**Mixed accuracy; large token savings against a constrained page-fetch agent.** Later tests found
no accuracy advantage over full reading or simple RAG.

## Early tool-use experiment

The baseline had a docs root and URL fetch, with no search engine, browser, local index, or context
compaction. Scull supplied compact lookups. These measure those interfaces, not a capable browsing
agent. Domains are anonymized; source-specific reports and transcripts remain excluded.

| Domain            | Task pass rate: Scull / page fetch | Agent tokens: Scull / page fetch |
| ----------------- | ---------------------------------: | -------------------------------: |
| Analytical engine |                      96.2% / 78.9% |                     1.9M / 41.4M |
| Developer CLI     |                      87.4% / 87.4% |                     2.2M / 44.6M |
| Niche language    |                      81.2% / 95.0% |                    13.9M / 31.2M |

The language artifact needed content fixes before the retained sweep. Compact facts did not
consistently help agents produce working code.

The September 5–6, 2026 ledger contains 304, 95, and 240 runs per condition respectively:
639 per condition, 1,278 total. The first domain uses equal-weighted model averages; the others
pool passes. Do not sum the displayed metrics across domains. [Method](https://caseyrfsmith.github.io/scull/docs/evaluation/methodology.md).

## Later full-read and RAG tests

Representative same-model results from separate decision-focused suites:

| Anonymized corpus          | Full read |   RAG | Scull | Agent tokens: full read / RAG / Scull |
| -------------------------- | --------: | ----: | ----: | ------------------------------------: |
| Reviewed API docs          |     12/12 | 12/12 | 12/12 |             211,397 / 95,940 / 69,145 |
| Synthetic adversarial docs |     10/10 | 10/10 | 10/10 |              15,261 / 16,407 / 40,761 |

No accuracy lift was observed across the broader four-model evaluation program; not every model
ran in every condition. A later size sweep found Scull did not beat RAG on the tested corpus.
Extra tool turns can erase smaller-result savings. These findings led development to stop.

## What remains worth testing

Compact local lookup may suit agents without web access or with small context budgets. The existing
results do not establish an advantage for private docs, local models, or air gaps. A next test should
compare the same local model and pinned docs through Scull and reasonable local retrieval, including
build/review effort and actual disconnected operation.