# Evaluation method

> Documentation index: [llms.min.txt](https://caseyrfsmith.github.io/scull/llms.min.txt) · Document manifest: [documents.json](https://caseyrfsmith.github.io/scull/documents.json)

Breadcrumbs: The experiment > Evaluation method

## Early constrained-agent runs

Same task prompts and models; different documentation interfaces:

| Condition  | Access                                                            |
| ---------- | ----------------------------------------------------------------- |
| Scull      | Compact ranked results and named lookups over precompiled schemas |
| Page fetch | Documentation root, URL fetch, and the task execution tool        |

The baseline had no search engine, browser, indexed retrieval, cleaned reader view, or context
compaction. Fetching returned up to 51,200 bytes with a 15-second timeout; content stayed in history.
Scull instructions encouraged targeted search, and some domains had concept-scoped retrieval.

These compare tool-and-prompt packages, not equally capable retrieval systems. The loop defaults
to 25 turns; individual runs can override it. Executable scorers checked outputs, not prose quality.

## Aggregation and cost

The September 5–6, 2026 runs cover 10 models and 1,278 task-condition runs. The analytical-domain
summary averages repeats within each model, then weights models equally. Other domains pool passes.
One API-error run was excluded rather than scored as a failure. Do not combine domains into one score.

Cost estimates cover agent inference only. They exclude extraction, review, indexing, embeddings,
maintenance, and infrastructure. Token savings are not total deployment savings.

## Later comparisons

Decision-focused tests added full-read and naive RAG conditions over small reviewed and synthetic
corpora. They found no accuracy advantage; a size sweep found conditional cost savings. These tests
answer a different question and must not be pooled with the early runs.

## Limits

Source domains are anonymized. Raw inputs, prompts, transcripts, and detailed reports are excluded;
this aggregate is not an independently reproducible benchmark. Early schemas were unreviewed model
extractions; failures drove corrections in one domain before its final sweep.

Results do not establish performance on private docs, local models, or air-gapped deployments.
No web access still allows strong local retrieval. [Results](https://caseyrfsmith.github.io/scull/docs/evaluation/results.md).