For agents: Skip to article. Documentation index. Read this page as Markdown.
Scull
Guide

Procedure: the pipeline

Run the extraction, index-building, and query stages in order.

Read as Markdown

Build → index → query. Three separate commands, three separate concerns.

1. Build schemas

set -a && source .env && set +a

scull-schemafier run \
  --concept myconcept \
  --docset /path/to/my-docset \
  --out schemas/ \
  --model baseten/zai-org/GLM-5.3-Flash

The conventional docset contains vocabulary.yaml, optional spec.json or openapi.{yaml,yml,json}, and optional prose/. Explicit input flags override discovered paths, and --out stays separate. Every discovered .md file is cleaned (stripped before the last H1, cut at footer sections), extracted, and reconciled against vocabulary.yaml. Spec is optional; an OpenAPI document skips the need for a hand-authored spec.json.

Output: schemas/myconcept.schema.yaml. Commit it. It carries schema_hash (content hash excluding the hash field itself) used by the KG builder for incremental rebuilds.

Spec and OAS provenance are recorded independently of emitted fields. This matters for response-only OAS documents and empty flat spec slices: sources.oas.hash or sources.spec.hash still lets the matching scull-schemafier check command detect whether the input changed. When both inputs are supplied, each keeps its own source record. When updating a concept with only --spec or --prose, the prior OAS structure block is preserved. A run that supplies --oas replaces the block, including clearing it when the new OAS has no structure. Required lists in generated OAS structures are sorted so identical input stays hash-stable across processes.

2. Build KG index

scull-schemafier kg build

Reads .schema.yaml files, embeds text via the configured provider, and publishes an immutable generation under kg/generations/. The kg/CURRENT pointer selects the complete generation containing vectors.npy, nodes.jsonl, partitions.json, meta.json, and the dedup report. Unchanged concepts reuse rows when their exact chunk fingerprint and provider identity match; older generations remain available for readers that already resolved them. Builds write v2, while old v1 indexes remain readable.

Force full rebuild if the embedding model or provider changes:

scull-schemafier kg build --force

Provider changes already trigger a full rebuild. Use --force to recover from damaged or unsupported index artifacts; different embedding spaces must never be mixed.

3. Query

No generative inference at query time for get / caveats / related / dedup. search calls Voyage AI embed() live for the query string only; it never generates an answer.

scull get myconcept field_name
scull caveats myconcept --severity high
scull search "retry backoff" --concepts job_runtime
scull related myconcept field_name
scull dedup --threshold 0.90

All commands work with core dependencies only. search needs [search] (voyageai/openai) or [local] (FastEmbed/MiniLM). V2 search memory-maps normalized float32 vectors and reads stored chunk text. Ordinary startup validates the catalog without parsing every YAML; exact schema commands load only requested concepts. scull validate still checks every artifact eagerly.