Schema sources and reconciliation

Scull compiles structured .schema.yaml files from one or more human-authored inputs at build time. At query time, everything is deterministic — same schema in, same answer out.

TL;DR: Start from whatever source gives you facts faster. OpenAPI specs give you field shapes for free — reserve LLM calls for the behavioral knowledge the spec can't express. Merge them with deterministic rules, flag disagreements, commit the output.

Source types

You mix and match freely. A single concept might have an OAS file, three prose pages, and a vocabulary section.

OAS-first workflow (recommended for APIs): Parse the OpenAPI spec into field candidates first, then run the LLM only on prose to get caveats and fill coverage gaps. This avoids generative extraction of fields already declared in the spec.

Input layout per concept

data/<concept>/
├── spec.yaml           # Optional: OpenAPI spec or spec.json
├── prose/
│   ├── guide.md
│   └── reference.md
└── vocabulary.yaml     # Or shared from repo-level vocabulary.yaml

Multiple sources are optional but recommended for accuracy. One source is fine if it's sufficient.

Pipeline stages

Stage 1: Extract

Each source type has its own extractor producing the same intermediate shape (kind: field or kind: caveat). No LLMs for spec/OAS — fully deterministic.

Extractors:

• oas_parser.py — Walk OpenAPI paths/operations backwards from endpoints; resolve $refs to schema names; build field_ids as {tag}.{SchemaName}.{prop} using operation tags for context. Also captures inline examples and constraint metadata.
• spec_extractor.py — Read flat JSON spec arrays
• prose_extractor.py — Clean markdown + send to LLM with vocabulary prompt

What the LLM extracts from prose:

• Caveats (behavioral surprises, severity-tagged)
• Decision rules ("use X instead of Y when Z")
• Coverage gaps (things mentioned in prose but absent from spec/OAS)
• Descriptions (enhanced meaning beyond type definitions)
• Anything the spec can't encode: timing, consistency guarantees, operational implications

The LLM does NOT extract from prose what's already in the spec: no need to ask an LLM "what is the amount field?" when OAS says type: number, required: true. The LLM's job shrinks from "extract everything" to "extract the why."

Stage 2: Reconcile

Fully deterministic merge. Takes field/caveat candidates from all extractors and produces the final .schema.yaml.

Field merging logic:

For each field_id (canonical name from vocabulary):

  1. Check which sources have this field
  2. Pick winner per property:
     ├─ value_type          ← Spec/OAS wins
     ├─ required            ← Spec/OAS wins
     ├─ default             ← Spec/OAS wins
     ├─ enum_values         ← Spec/OAS wins
     ├─ description         ← Prose always wins silently
     └─ minimum/maximum/etc ← Spec/OAS wins (constraints)

  3. On conflict between spec and prose on type/required/default/enum:
     ├─ Write the spec value to the output
     ├─ Add to conflicts[] block
     └─ Set needs_review: true

  4. Confidence scoring:
     ├─ Both sources present → high
     ├─ Only spec/OAS → medium
     └─ Only prose → medium

  5. Audit the complete reconciled schema:
     ├─ Field absent from vocabulary → field_assignment_not_in_vocabulary
     ├─ Prose matches a reviewer-owned assignment_excludes phrase
     │  → suspicious_field_assignment
     └─ Preserve the field unchanged and set needs_review: true

Vocabulary entries may include narrow, reviewer-owned exclusions for known semantic collisions:

- canonical: procedure_declaration
  aliases: [procedure, proc]
  assignment_excludes: [macro, closure, operator overload]

The exclusions are included in the extraction prompt and checked deterministically against the final schema, including fields carried over from an earlier run. They are not inferred from alias similarity and never cause automatic reassignment. Audit a committed schemaset without an LLM or file mutation using:

scull-schemafier audit-assignments --schemas schemas --vocab vocabulary.yaml

Coverage gap detection (the diff you need to watch for):

  ┌─────────────────────────────┬──────────────────────────────┐
  │ Field in spec               │ Field in prose but not in    │
  │ but not in prose            │ spec                         │
  ├─────────────────────────────┼──────────────────────────────┤
  │ "Docs don't discuss this"   │ "Unspecified feature"        │
  │ Warning: spec_coverage_gap  │ Warning: coverage_overlap    │
  │ Operator may not know this  │ Docs describe behavior       │
  │ field exists                │ outside spec definition      │
  └─────────────────────────────┴──────────────────────────────┘

These warnings go into the schema's warnings[] block. Structural conflicts and extraction failures set needs_review: true; informational notices remain visible without blocking acceptance.

Caveat merging logic:

  For each caveat from prose:
    ├─ Compute evidence_hash = sha256(evidence_text)[:12]
    ├─ Compare against stored caveat from previous run
    ├─ If hash changed: emit caveat_evidence_changed warning, needs_review: true
    └─ Otherwise: accept as-is (caveats come ONLY from prose, never from spec/OAS)

Stage 3: Index (KG builder)

Incremental rebuild. Skips schemas whose schema_hash hasn't changed. Embeds text content and builds graph edges. Zero LLM calls — reads from .schema.yaml only.

Conflict and warning blocks

These are the reconciliation signals — what gets written when sources disagree or coverage is incomplete.

conflicts

Recorded when spec/OAS and prose disagree on structural properties:

conflicts:
  - property: value_type
    field_id: amount
    spec_value: integer
    prose_value: string
  - property: required
    field_id: metadata.name
    spec_value: true
    prose_value: false

Only these four properties produce conflicts: value_type, required, default, enum_values. Description differences never trigger conflicts — prose always wins those silently.

On conflict, the spec/OAS value is written to the output and the record goes into conflicts[] for human review. The spec won. The human checks whether it should have been.

warnings

Warnings are persisted so an operator can see what happened during extraction. extraction_failure and empty_candidates mean a source was attempted but produced no usable output. Those warnings, field_assignment_not_in_vocabulary, and suspicious_field_assignment set needs_review: true and make the CLI exit with code 2. Notices such as thin_source_skipped, no_prose_files, and source_missing remain non-blocking.

warnings:
  - kind: extraction_failure
    message: "LLM call failed for prose/reference.md"
  - kind: unmapped_spec_field
    field_name: legacy_timeout
    message: "No vocabulary entry found"
  - kind: suspicious_field_assignment
    field_id: procedure_declaration
    assignment_excludes: [macro]
    message: "Prose-backed field matches a reviewer-owned exclusion"
  - kind: caveat_evidence_changed
    caveat_id: payload-size-limit
    old_hash: be81b818094f
    new_hash: 5b81b818094f
    message: "Evidence drifted on re-run"
  - kind: source_missing
    url: https://example.com/doc-page
    message: "URL unreachable during re-fetch"
  - kind: spec_coverage_gap
    field_id: undocumented_field
    message: "Field exists in spec but prose doesn't mention it"
  - kind: coverage_overlap
    field_id: undocumented_feature
    message: "Prose describes behavior not in spec"

Output format reference

See docs/schema-format.md for the full field-by-field spec of .schema.yaml structure. Key structural points:

schema_version: '0.1'
concept: order_api
sources:
  spec:                    # Single OAS or spec file
    url: data/order_api/spec.yaml
    hash: <sha256>
  prose:                   # Multiple prose files accepted
    - url: data/order_api/prose/lifecycle.md
    - url: data/order_api/prose/statuses.md

fields:
  order_id:
    value_type: string
    required: true
    description: "Unique transaction identifier..."
    confidence: high          # Both spec and prose present
    sources: [spec, prose]

conflicts: []                 # Empty = no disagreements
warnings: []                  # Empty = nothing noteworthy
needs_review: false           # Conflicts/failures block; informational notices do not
deprecated: false
schema_hash: <sha256>         # Trigger for KG incremental rebuild
_meta:
  generated_at: '2026-09-04T15:51:40Z'

Extraction work by source type

Call counts and cost depend on input length, batching, coverage passes, retries, fallback models, and review. No fixed per-page savings are established. Structured input avoids asking a model to infer declared types, but source quality and parser coverage still need checking.

Common patterns

API product with an OpenAPI spec

# Step 1: Download/copy the OpenAPI spec locally
cp /path/to/openapi.yaml data/order_api/openapi.yaml

# Step 2: Run scull-schemafier with OAS + prose
scull-schemafier run \
  --concept order_api \
  --oas data/order_api/openapi.yaml \
  --prose data/order_api/prose/ \
  --vocab vocabulary.yaml \
  --out schemas/

# Step 3: Review conflicts (should be minimal since OAS is ground truth)
grep -A 5 "^conflicts:" schemas/order_api.schema.yaml

# Step 4: Build index
scull-schemafier kg build

Internal service with no public API docs

# Use spec.json directly — you author the field list yourself
scull-schemafier run \
  --concept my_service \
  --spec data/my_service/spec.json \
  --prose data/my_service/prose/ \
  --out schemas/my_service.schema.yaml

Concept documented only in prose (no formal spec)

# LLM handles everything from docs — acceptable for smaller concepts
scull-schemafier run \
  --concept boarding_flow \
  --prose data/boarding/ \
  --out schemas/boarding_flow.schema.yaml

Maintenance and re-runs

When upstream docs change, run scull-schemafier run again with the same arguments. The reconciler will:

1. Re-extract changed prose files (or all prose with --force); parse supplied spec/OAS inputs fresh
2. Diff new candidates against stored schema
3. Emit updated conflicts[] and warnings[] if anything shifted
4. Produce a new schema_hash
5. Set needs_review: true if a conflict, evidence drift, extraction failure, empty extraction, or assignment-audit finding is present

On an unchanged rerun, prose files are skipped by source hash. Their fields and caveats are carried forward unchanged, and a fresh spec/OAS candidate for one of those fields receives the previous prose description and locator while keeping the current structural values. The resulting semantic content and schema_hash stay stable across repeated cache hits. A prior needs_review: true also remains visible while its source is skipped; rerunning extraction is what allows that state to be recomputed.

CI staleness check (scull-schemafier check) compares source hashes against stored values — if a prose URL's content changed but your schema wasn't regenerated, it flags it. Re-run the full pipeline to pick up changes.

Why this matters for Scull's design

This reconciliation layer is what makes Scull different from RAG. A RAG system dumps raw docs into a vector store and hopes retrieval surfaces the right chunk. Scull's reconciler actively checks: do the docs agree with the spec? Are there fields no one talked about? Is evidence drifting? It writes those findings into a human-readable YAML file you can diff in a PR.

The operator reviews the schema, approves the conflicts, and commits the result. Query time never touches docs, never reruns LLMs, never hallucinates. All the work was done deterministically or reviewed-by-a-human at build time.