Schema file format

A .schema.yaml file is the output of one pipeline run on one concept. It's the contract between the scull-schemafier and the CLI. The format is stable and human-reviewable — you're meant to read and diff these files.

Top-level structure

schema_version: "0.1"
concept: caveat
schema_hash: 3306230a76d71d81...    # SHA-256 of all content fields
_meta:
  generated_at: "2026-09-04T15:51:40Z"
  review:
    reviewer: reviewer@example.com
    note: Verified fields, evidence, attachments, and retained diagnostics
    approved_at: "2026-09-09T18:30:00+00:00"
    fingerprint: 5b9a6f...

sources:
  spec:
    url: data/caveat/spec.json
    hash: a7b8c74d...
    status: ok
  prose:
    - url: https://docs.example.com/caveat-format
      hash: 78235a9a...
      status: ok

fields:
  id:
    value_type: string
    required: true
    default: null
    enum_values: null
    description: "Stable kebab-case identifier derived from..."
    confidence: high
    sources: [spec, prose]
    locator:
      url: https://docs.example.com/caveat-format
      heading: Caveat identifiers

caveats:
  - id: id-never-changes-automatically
    summary: "Caveat IDs are deterministic but never automatically updated..."
    severity: medium
    sources: [prose]
    confidence: high
    evidence: "The ID is derived once at creation time and stored..."
    evidence_hash: 17baf96c0493
    applies_to: [id]
    applies_when: [operation.update]
    locator:
      url: https://docs.example.com/caveat-format
      heading: Caveat identifiers

structure:          # Full type hierarchy from OpenAPI specs — walks endpoints backwards
  bill.BillOutData:   # {tag}.{SchemaName} — tag comes from operation tags
    type: object
    required: [billNumber, netAmount, billDate]
    properties:
      billNumber:
        type: string
        minLength: 1
        maxLength: 50
      netAmount:
        type: number
        minimum: 0

needs_review: false
conflicts: []
warnings: []
deprecated: false

Field reference

schema_version

Format version. Readers accept "0.1" and "0.2"; the CLI validates this on load. JSON Schema constraint artifacts require "0.2". Older readers that only support "0.1" reject them rather than silently ignoring their constraints.

constraint_bundle (format 0.2)

A reference to a digest-bound closure manifest under constraints/, with each original JSON resource stored once by content digest. Distribute that directory alongside the .schema.yaml files. Readers verify paths and hashes locally; no source checkout or network is required. Missing or altered resources are errors.

Interpretation records distinguish supported rules, annotations, and rules preserved without interpretation. Partial coverage stays partial after review approval. Use scull inspect-schema <concept> <resource-uri> [json-pointer] --limit 100 to inspect original rules, interpretation status, and reference links. get and context --detail expose complete scoped records; default context reduces them to concise constraints, requiredness, completeness, and inspection links. Ambiguous property aliases retain their distinct rule identities and candidates.

Property fields may also expose default, example_values, and runtime_transforms. Defaults and examples are JSON Schema annotations. runtime_transforms records application-specific behavior declared by the source (for example trimming a string); it is surfaced for authoring but remains a nonstandard keyword, so the constraint bundle does not claim that ordinary JSON Schema validation reproduces it.

concept

Canonical concept name. Must match a top-level key in vocabulary.yaml. This is what you pass to scull get, scull caveats, etc.

schema_hash

SHA-256 of stable content fields, excluding schema_hash, _meta, warnings, and conflicts. Warnings may reflect non-deterministic extraction diagnostics, while conflicts are derived from already-hashed fields. scull validate recomputes and compares this hash. The hash detects accidental content mismatch; it is not a signature and does not prove authenticity or correctness.

This is the primary change signal in the system:

• KG builder reuses a concept's rows when its provider identity, chunker version, and exact embedded-text fingerprint are unchanged
• scull-schemafier check (CI staleness) compares source hashes, not schema_hash — those are two different signals

Deliberately not based on generated_at. Regenerating with no content changes produces the same hash — nothing downstream re-processes.

_meta

Non-content metadata. generated_at is an ISO 8601 timestamp. An approved artifact also has a review mapping containing non-empty reviewer, note, approved_at, and fingerprint strings. finalize --approve creates that record.

The review fingerprint binds the schema's content together with its current warnings and conflicts, excluding only _meta and schema_hash. Any later edit to content, diagnostics, or review state invalidates release approval until a reviewer approves the new state. The fingerprint records what was reviewed; it is not a cryptographic identity or a semantic validation result.

sources

Hashed references to every input consumed during the pipeline run.

sources.spec:

• url — path or URL of the spec JSON file
• hash — SHA-256 of the file bytes at extraction time
• status — ok, missing, or stale

sources.prose[]:

• url — canonical URL (from --source-url flag, or local file path as fallback)
• hash — SHA-256 of the file bytes at extraction time
• status — ok, missing, or stale
• signals — metadata mapped from YAML frontmatter via scull.yaml metadata_mappings. Common signals: source_url, last_verified, owner.

status: missing means the source URL was unreachable on the most recent re-fetch. The schema and its content are preserved — a missing page may be temporary. deprecated: true is a separate, human-set flag.

fields

Map of canonical field name → field entry. Keyed by the canonical name from vocabulary.yaml, not whatever the source happens to call it.

Field entry properties:

Reconciler rules in plain English:

• Spec wins on everything structural: value_type, required, default, enum_values
• Prose wins on everything behavioral: description and all caveats
• Description differences between spec and prose are not conflicts — prose always wins silently
• Structural disagreements between spec and prose → conflicts[] entry + needs_review: true

caveats

List of behavioral caveat entries. Always extracted from prose — the spec never emits caveats.

Caveats are the core value of Scull for agent use. A caveat is anything that would surprise a developer: a non-obvious constraint, a silent failure mode, an exception to the general rule, behavior that contradicts intuition.

Caveat entry properties:

applies_when refines field-level applicability without semantic inference. Tags form a hierarchy: a request for consume.volume.subpath also matches the broader consume.volume tag. Multiple tags on one caveat are alternative situations. The scull-schemafier does not invent these tags; a reviewer adds them to a committed schema, and reconciliation preserves them when the caveat keeps its stable ID. Calling scull context without situation tags remains conservative and returns the complete set.

Severity scale:

ID stability: caveat IDs are derived from the first 5 significant words of the summary plus a 6-hex SHA-256 suffix. They're stable across re-runs even if the LLM rephrases slightly. To change an ID, edit it directly in the schema file — it will not be automatically regenerated.

Evidence drift: on re-runs, if the LLM quotes slightly different text than last time, evidence_hash changes and a caveat_evidence_changed warning is emitted with needs_review: true. This is expected behavior for re-runs where the source docs didn't change — accept it after verifying the evidence is still accurate.

When an unchanged prose file is skipped, its existing fields and caveats are carried forward without an execution-state marker. If a current spec/OAS run re-emits one of those fields, its prior prose description and locator are restored while the current structural values remain authoritative. This makes the first cached rerun and later cached reruns preserve both content and schema_hash.

needs_review

true when the reconciler found at least one:

• Conflict between spec and prose on value_type, required, default, or enum_values
• caveat_evidence_changed warning (evidence hash drifted from stored value)
• extraction_failure or empty_candidates warning that might have omitted content

Unresolved diagnostics tied to a skipped source (or not safely attributable to a source) remain visible during cache hits. A failure tied to a successfully reprocessed source can clear, even if another source was skipped in the same run. Informational notices such as thin_source_skipped do not set this flag by themselves.

The schema is still written and usable for development. The scull-schemafier exits with code 2. CI pipelines should treat exit code 2 as blocking. A reviewer can correct the artifact or explicitly accept its current warnings/conflicts with finalize --approve; release validation then verifies that the recorded decision still matches the artifact.

conflicts

List of cross-source disagreements on structural properties. Spec wins in the output, but the conflict is recorded for human review:

conflicts:
  - property: value_type
    field_id: threshold
    spec_value: integer
    prose_value: float

Only value_type, required, default, and enum_values produce conflicts. Description differences do not.

warnings

Extractor and reconciler warnings. Nothing is silently dropped — if something went wrong during extraction, it appears here.

deprecated

true only when upstream spec or prose explicitly marks the concept deprecated. This is a deliberate human action — not automatically set when a source goes missing or a URL returns 404.

Structure block (OAS ingestion)

When using OAS-based schema generation (scull-schemafier run --oas <file>), a structure: block captures the full extracted type hierarchy from the OpenAPI spec. It's a flat dict keyed by (optionally normalized — see below) schema name, not nested by endpoint tag:

structure:
  CreateOrderRequest:          # Schema name resolved via $ref (post-normalization)
    type: object
    required: [amount, currency]
    properties:
      amount: integer
      ftype: string
    endpoint: POST /orders
    examples:
      CreateOrderRequest.amount: [50000]

Every structure entry — whether it came from a $ref-derived body, an inline (non-$ref) body, or a response body — is written in this same flat shape (type/properties/required directly alongside endpoint/examples, plus response_code for response-derived entries). scull shape also transparently normalizes the older {"flattened": {...}}-wrapped shape at display time, for any schema file written by a version of the parser before this was unified — but the parser itself never produces that shape anymore.

Field naming (in the fields: block, not structure:) is deterministic — {tag}.{SchemaName}.{prop} comes straight from the OAS operation's tags and $ref resolution. No manual mapping needed; group resolution happens through tree-walking endpoints where tags and operation metadata provide context. The parser resolves all $ref chains with cycle detection (including two fields that reference the same shared schema — that's not a cycle), flattens allOf composition, collapses oneOf[string, null] into string, and follows enum schemas through refs.

The structure: block enables standalone JSON payload lookups without needing an external OpenAPI service. Agents can query this directly via scull shape <concept> [schema_name] (zero inference, same as get/caveats) to get exact request/response shapes, while the fields/caveats layer answers behavioral questions about when to use X vs Y and what breaks silently.

Schema name normalization

For APIs with noisy or verbose schema naming (e.g., QueryPayoutRecordsItem, TransferOutDetailVendor), you can configure regex normalization rules in scull.yaml:

oas_normalize:
  rules:
    - match: "(?i)(Query|Result|Search).*Record"
      rename: "\\1"
    - match: "(.*)DetailRecord$"
      rename: "\\1.Detail"

Rules apply in order; each schema name matches at most the first rule that fires and the result is used for both fields: field_ids and structure: keys. There is no separate persistence step or __oas_mappings__ block—rules are applied fresh from scull.yaml on every run, so a rule change takes effect immediately on the next scull-schemafier run --oas. No code path reads or writes __oas_mappings__ today; if that key appears in an older schema file, it is inert and safe to delete. When no rules are configured, raw schema names pass through unchanged—no assumptions, no forced grouping.