# Procedure: the pipeline

> Documentation index: [llms.min.txt](https://caseyrfsmith.github.io/scull/llms.min.txt) · Document manifest: [documents.json](https://caseyrfsmith.github.io/scull/documents.json)

Breadcrumbs: Build and use > Procedure: the pipeline

Build → index → query. Three separate commands, three separate concerns.

## 1. Build schemas

```bash
set -a && source .env && set +a

scull-schemafier run \
  --concept myconcept \
  --docset /path/to/my-docset \
  --out schemas/ \
  --model baseten/zai-org/GLM-5.3-Flash
```

The conventional docset contains `vocabulary.yaml`, optional `spec.json` or
`openapi.{yaml,yml,json}`, and optional `prose/`. Explicit input flags override discovered paths,
and `--out` stays separate. Every discovered `.md` file is cleaned (stripped before the last H1,
cut at footer sections), extracted, and reconciled against `vocabulary.yaml`. Spec is optional;
an OpenAPI document skips the need for a hand-authored `spec.json`.

Output: `schemas/myconcept.schema.yaml`. Commit it. It carries `schema_hash` (content hash excluding the hash field itself) used by the KG builder for incremental rebuilds.

Spec and OAS provenance are recorded independently of emitted fields. This matters for response-only
OAS documents and empty flat spec slices: `sources.oas.hash` or `sources.spec.hash` still lets the
matching `scull-schemafier check` command detect whether the input changed. When both inputs are supplied,
each keeps its own source record. When updating a concept with only `--spec` or `--prose`, the prior
OAS `structure` block is preserved. A run that supplies `--oas` replaces the block, including clearing
it when the new OAS has no structure. Required lists in generated OAS structures are sorted so
identical input stays hash-stable across processes.

## 2. Build KG index

```bash
scull-schemafier kg build
```

Reads `.schema.yaml` files, embeds text via the configured provider, and publishes an immutable generation
under `kg/generations/`. The `kg/CURRENT` pointer selects the complete generation containing
`vectors.npy`, `nodes.jsonl`, `partitions.json`, `meta.json`, and the dedup report. Unchanged
concepts reuse rows when their exact chunk fingerprint and provider identity match; older
generations remain available for readers that already resolved them. Builds write v2, while old v1
indexes remain readable.

Force full rebuild if the embedding model or provider changes:

```bash
scull-schemafier kg build --force
```

Provider changes already trigger a full rebuild. Use `--force` to recover from damaged or
unsupported index artifacts; different embedding spaces must never be mixed.

## 3. Query

No generative inference at query time for `get` / `caveats` / `related` / `dedup`. `search` calls Voyage AI `embed()` live for the query string only; it never generates an answer.

```bash
scull get myconcept field_name
scull caveats myconcept --severity high
scull search "retry backoff" --concepts job_runtime
scull related myconcept field_name
scull dedup --threshold 0.90
```

All commands work with core dependencies only. `search` needs `[search]` (`voyageai`/`openai`) or
`[local]` (FastEmbed/MiniLM). V2 search memory-maps normalized float32 vectors and reads stored chunk
text. Ordinary startup validates the catalog without parsing every YAML; exact schema commands load
only requested concepts. `scull validate` still checks every artifact eagerly.