# Schema sources and reconciliation

> Documentation index: [llms.min.txt](https://caseyrfsmith.github.io/scull/llms.min.txt) · Document manifest: [documents.json](https://caseyrfsmith.github.io/scull/documents.json)

Breadcrumbs: Reference > Schema sources and reconciliation

Scull compiles structured `.schema.yaml` files from one or more human-authored inputs at build time. At query time, everything is deterministic — same schema in, same answer out.

**TL;DR:** Start from whatever source gives you facts faster. OpenAPI specs give you field shapes for free — reserve LLM calls for the behavioral knowledge the spec can't express. Merge them with deterministic rules, flag disagreements, commit the output.

## Source types

| Type                               | What it gives you                                                                                                                  | When to use it                                                                      |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| **OpenAPI/OAS (`*.yaml`)**         | Endpoint signatures, request/response field shapes, exact types, required lists, enum values, constraints (`minLength`, `maximum`) | API-driven products where specs are authoritative                                   |
| **Spec JSON (`spec.json`)**        | Flat array of `{name, value_type, required, description}` slices you authored yourself                                             | Legacy pipelines, concepts without OAS, or when you already have a clean spec slice |
| **Prose (`prose/*.md`)**           | Behavioral rules, decision guides, caveats ("why this exists", silent failure modes, best practices)                               | Always useful — every concept benefits from prose-contextualized caveats            |
| **Vocabulary (`vocabulary.yaml`)** | Canonical field names + aliases. Join key for all extractors.                                                                      | Prerequisite — always needed                                                        |

You mix and match freely. A single concept might have an OAS file, three prose pages, and a vocabulary section.

**OAS-first workflow** (recommended for APIs): Parse the OpenAPI spec into field candidates first, then run the LLM only on prose to get caveats and fill coverage gaps. This avoids generative extraction of fields already declared in the spec.

## Input layout per concept

```
data/<concept>/
├── spec.yaml           # Optional: OpenAPI spec or spec.json
├── prose/
│   ├── guide.md
│   └── reference.md
└── vocabulary.yaml     # Or shared from repo-level vocabulary.yaml
```

Multiple sources are optional but recommended for accuracy. One source is fine if it's sufficient.

## Pipeline stages

### Stage 1: Extract

Each source type has its own extractor producing the same intermediate shape (`kind: field` or `kind: caveat`). No LLMs for spec/OAS — fully deterministic.

**Extractors:**

- `oas_parser.py` — Walk OpenAPI paths/operations backwards from endpoints; resolve `$ref`s to schema names; build field\_ids as `{tag}.{SchemaName}.{prop}` using operation tags for context. Also captures inline examples and constraint metadata.
- `spec_extractor.py` — Read flat JSON spec arrays
- `prose_extractor.py` — Clean markdown + send to LLM with vocabulary prompt

**What the LLM extracts from prose:**

- Caveats (behavioral surprises, severity-tagged)
- Decision rules ("use X instead of Y when Z")
- Coverage gaps (things mentioned in prose but absent from spec/OAS)
- Descriptions (enhanced meaning beyond type definitions)
- Anything the spec can't encode: timing, consistency guarantees, operational implications

**The LLM does NOT extract from prose what's already in the spec:** no need to ask an LLM "what is the `amount` field?" when OAS says `type: number, required: true`. The LLM's job shrinks from "extract everything" to "extract the why."

### Stage 2: Reconcile

Fully deterministic merge. Takes field/caveat candidates from all extractors and produces the final `.schema.yaml`.

**Field merging logic:**

```
For each field_id (canonical name from vocabulary):

  1. Check which sources have this field
  2. Pick winner per property:
     ├─ value_type          ← Spec/OAS wins
     ├─ required            ← Spec/OAS wins
     ├─ default             ← Spec/OAS wins
     ├─ enum_values         ← Spec/OAS wins
     ├─ description         ← Prose always wins silently
     └─ minimum/maximum/etc ← Spec/OAS wins (constraints)

  3. On conflict between spec and prose on type/required/default/enum:
     ├─ Write the spec value to the output
     ├─ Add to conflicts[] block
     └─ Set needs_review: true

  4. Confidence scoring:
     ├─ Both sources present → high
     ├─ Only spec/OAS → medium
     └─ Only prose → medium

  5. Audit the complete reconciled schema:
     ├─ Field absent from vocabulary → field_assignment_not_in_vocabulary
     ├─ Prose matches a reviewer-owned assignment_excludes phrase
     │  → suspicious_field_assignment
     └─ Preserve the field unchanged and set needs_review: true
```

Vocabulary entries may include narrow, reviewer-owned exclusions for known semantic collisions:

```yaml
- canonical: procedure_declaration
  aliases: [procedure, proc]
  assignment_excludes: [macro, closure, operator overload]
```

The exclusions are included in the extraction prompt and checked deterministically against the
final schema, including fields carried over from an earlier run. They are not inferred from alias
similarity and never cause automatic reassignment. Audit a committed schemaset without an LLM or
file mutation using:

```bash
scull-schemafier audit-assignments --schemas schemas --vocab vocabulary.yaml
```

**Coverage gap detection** (the diff you need to watch for):

```
  ┌─────────────────────────────┬──────────────────────────────┐
  │ Field in spec               │ Field in prose but not in    │
  │ but not in prose            │ spec                         │
  ├─────────────────────────────┼──────────────────────────────┤
  │ "Docs don't discuss this"   │ "Unspecified feature"        │
  │ Warning: spec_coverage_gap  │ Warning: coverage_overlap    │
  │ Operator may not know this  │ Docs describe behavior       │
  │ field exists                │ outside spec definition      │
  └─────────────────────────────┴──────────────────────────────┘
```

These warnings go into the schema's `warnings[]` block. Structural conflicts and extraction failures
set `needs_review: true`; informational notices remain visible without blocking acceptance.

**Caveat merging logic:**

```
  For each caveat from prose:
    ├─ Compute evidence_hash = sha256(evidence_text)[:12]
    ├─ Compare against stored caveat from previous run
    ├─ If hash changed: emit caveat_evidence_changed warning, needs_review: true
    └─ Otherwise: accept as-is (caveats come ONLY from prose, never from spec/OAS)
```

### Stage 3: Index (KG builder)

Incremental rebuild. Skips schemas whose `schema_hash` hasn't changed. Embeds text content and builds graph edges. Zero LLM calls — reads from `.schema.yaml` only.

## Conflict and warning blocks

These are the reconciliation signals — what gets written when sources disagree or coverage is incomplete.

### `conflicts`

Recorded when spec/OAS and prose disagree on structural properties:

```yaml
conflicts:
  - property: value_type
    field_id: amount
    spec_value: integer
    prose_value: string
  - property: required
    field_id: metadata.name
    spec_value: true
    prose_value: false
```

**Only these four properties produce conflicts:** `value_type`, `required`, `default`, `enum_values`. Description differences never trigger conflicts — prose always wins those silently.

On conflict, the spec/OAS value is written to the output and the record goes into `conflicts[]` for human review. The spec won. The human checks whether it should have been.

### `warnings`

Warnings are persisted so an operator can see what happened during extraction. `extraction_failure`
and `empty_candidates` mean a source was attempted but produced no usable output. Those warnings,
`field_assignment_not_in_vocabulary`, and `suspicious_field_assignment` set `needs_review: true`
and make the CLI exit with code 2. Notices such as `thin_source_skipped`, `no_prose_files`, and
`source_missing` remain non-blocking.

```yaml
warnings:
  - kind: extraction_failure
    message: "LLM call failed for prose/reference.md"
  - kind: unmapped_spec_field
    field_name: legacy_timeout
    message: "No vocabulary entry found"
  - kind: suspicious_field_assignment
    field_id: procedure_declaration
    assignment_excludes: [macro]
    message: "Prose-backed field matches a reviewer-owned exclusion"
  - kind: caveat_evidence_changed
    caveat_id: payload-size-limit
    old_hash: be81b818094f
    new_hash: 5b81b818094f
    message: "Evidence drifted on re-run"
  - kind: source_missing
    url: https://example.com/doc-page
    message: "URL unreachable during re-fetch"
  - kind: spec_coverage_gap
    field_id: undocumented_field
    message: "Field exists in spec but prose doesn't mention it"
  - kind: coverage_overlap
    field_id: undocumented_feature
    message: "Prose describes behavior not in spec"
```

## Output format reference

See `docs/schema-format.md` for the full field-by-field spec of `.schema.yaml` structure. Key structural points:

```yaml
schema_version: '0.1'
concept: order_api
sources:
  spec:                    # Single OAS or spec file
    url: data/order_api/spec.yaml
    hash: <sha256>
  prose:                   # Multiple prose files accepted
    - url: data/order_api/prose/lifecycle.md
    - url: data/order_api/prose/statuses.md

fields:
  order_id:
    value_type: string
    required: true
    description: "Unique transaction identifier..."
    confidence: high          # Both spec and prose present
    sources: [spec, prose]

conflicts: []                 # Empty = no disagreements
warnings: []                  # Empty = nothing noteworthy
needs_review: false           # Conflicts/failures block; informational notices do not
deprecated: false
schema_hash: <sha256>         # Trigger for KG incremental rebuild
_meta:
  generated_at: '2026-09-04T15:51:40Z'
```

## Extraction work by source type

| Inputs          | Deterministic work                          | Generative work                      |
| --------------- | ------------------------------------------- | ------------------------------------ |
| Prose only      | Vocabulary checks and reconciliation        | Draft fields and caveats             |
| Spec + prose    | Parse declared fields and reconcile         | Draft behavioral details and caveats |
| OpenAPI + prose | Parse supported API structure and reconcile | Draft behavioral details and caveats |

Call counts and cost depend on input length, batching, coverage passes, retries, fallback models,
and review. No fixed per-page savings are established. Structured input avoids asking a model to
infer declared types, but source quality and parser coverage still need checking.

## Common patterns

### API product with an OpenAPI spec

```bash
# Step 1: Download/copy the OpenAPI spec locally
cp /path/to/openapi.yaml data/order_api/openapi.yaml

# Step 2: Run scull-schemafier with OAS + prose
scull-schemafier run \
  --concept order_api \
  --oas data/order_api/openapi.yaml \
  --prose data/order_api/prose/ \
  --vocab vocabulary.yaml \
  --out schemas/

# Step 3: Review conflicts (should be minimal since OAS is ground truth)
grep -A 5 "^conflicts:" schemas/order_api.schema.yaml

# Step 4: Build index
scull-schemafier kg build
```

### Internal service with no public API docs

```bash
# Use spec.json directly — you author the field list yourself
scull-schemafier run \
  --concept my_service \
  --spec data/my_service/spec.json \
  --prose data/my_service/prose/ \
  --out schemas/my_service.schema.yaml
```

### Concept documented only in prose (no formal spec)

```bash
# LLM handles everything from docs — acceptable for smaller concepts
scull-schemafier run \
  --concept boarding_flow \
  --prose data/boarding/ \
  --out schemas/boarding_flow.schema.yaml
```

## Maintenance and re-runs

When upstream docs change, run `scull-schemafier run` again with the same arguments. The reconciler will:

1. Re-extract changed prose files (or all prose with `--force`); parse supplied spec/OAS inputs fresh
2. Diff new candidates against stored schema
3. Emit updated `conflicts[]` and `warnings[]` if anything shifted
4. Produce a new `schema_hash`
5. Set `needs_review: true` if a conflict, evidence drift, extraction failure, empty extraction, or assignment-audit finding is present

On an unchanged rerun, prose files are skipped by source hash. Their fields and caveats are carried
forward unchanged, and a fresh spec/OAS candidate for one of those fields receives the previous
prose description and locator while keeping the current structural values. The resulting semantic
content and `schema_hash` stay stable across repeated cache hits. A prior `needs_review: true` also
remains visible while its source is skipped; rerunning extraction is what allows that state to be
recomputed.

CI staleness check (`scull-schemafier check`) compares source hashes against stored values — if a prose URL's content changed but your schema wasn't regenerated, it flags it. Re-run the full pipeline to pick up changes.

## Why this matters for Scull's design

This reconciliation layer is what makes Scull different from RAG. A RAG system dumps raw docs into a vector store and hopes retrieval surfaces the right chunk. Scull's reconciler actively checks: do the docs agree with the spec? Are there fields no one talked about? Is evidence drifting? It writes those findings into a human-readable YAML file you can diff in a PR.

The operator reviews the schema, approves the conflicts, and commits the result. Query time never touches docs, never reruns LLMs, never hallucinates. All the work was done deterministically or reviewed-by-a-human at build time.