# Procedure: KG and search

> Documentation index: [llms.min.txt](https://caseyrfsmith.github.io/scull/llms.min.txt) · Document manifest: [documents.json](https://caseyrfsmith.github.io/scull/documents.json)

Breadcrumbs: Build and use > Procedure: KG and search

`kg_builder` is a separate step from query. It reads `.schema.yaml`, not raw docs.

## Build

```bash
scull-schemafier kg build
```

Reads `schemas/*.schema.yaml`. Only changed `schema_hash` entries are re-embedded. Produces:

- `kg/CURRENT`, an atomic pointer to the complete published generation
- `kg/generations/<generation>/vectors.npy` (one normalized float32 matrix)
- `kg/generations/<generation>/nodes.jsonl` (ordered node metadata, exact embedded text, relationships)
- `kg/generations/<generation>/partitions.json` (concept row ranges)
- `kg/generations/<generation>/meta.json` (format/provider identity and schema catalog)
- `kg/generations/<generation>/dedup_report.json`

Builds publish v2 only. Existing v1 generation or flat indexes remain readable until rebuilt.
Unchanged concepts reuse vector rows when their provider, chunker version, and chunk fingerprint
match; the builder still stages and validates a complete consolidated generation.

For a release build, require the complete reviewed-artifact contract:

```bash
scull validate --release
scull-schemafier kg build --release --schemas schemas --out kg
```

`--release` requires a non-empty schemaset, valid structure and attachments, current hashes,
`needs_review: false`, and an approval fingerprint matching every artifact's current content and
diagnostics. It does not perform semantic review. Ordinary builds remain available for development.

Run a build after changing the embedding model or provider (`voyage` → `openai` or `local`). A changed
identity automatically forces re-embedding; `--force` also explicitly rebuilds everything:

```bash
scull-schemafier kg build --force
```

Failed builds leave the previous generation intact. Existing flat indexes remain readable;
the next build publishes v2. Readers pin one generation per operation, and
old generations remain on disk for readers already using them. Distribute `CURRENT` and its
complete referenced generation, including normally git-ignored vectors. See the
[KG reference](https://caseyrfsmith.github.io/scull/docs/reference/kg-builder.md) for retention details.

Deploy `schemas/` and that index snapshot as one revision, then recreate or reload long-lived
providers. The v2 catalog binds schema filenames, raw artifact digests, and semantic hashes before query
embedding and rejects a mixed deployment. Roll back both schemas and index together; restoring
only one side intentionally fails this compatibility check.

Ordinary CLI/provider startup uses the catalog without parsing every YAML file; an exact lookup
loads and validates only its concept. `scull validate` remains the full eager schemaset check.
Search memory-maps the matrix read-only. Dedup similarity uses bounded blocks. If validation reports
a corrupt or unsupported index, rebuild with `scull-schemafier kg build --force`.

## Visualize

```bash
python tools/kg_viz.py
```

Opens `kg/graph.html` — D3 force graph, self-contained. Nodes: concept (indigo), field (emerald), caveat (amber). Hover for summaries.

## Hybrid search

`scull search` computes:

```
final_score = cosine_similarity(query_vec, node_vec) + provider_lexical_weight * lexical_overlap_ratio
```

Lexical overlap matches significant terms (3+ chars, minus stopwords, plus operators like `=>`, `|>`). Prevents syntax queries (e.g., "pipe operator") from being buried under noise.

Search needs `kg/` built. Without it, `scull search` reports "Run 'scull-schemafier kg build' first."