For agents: Skip to article. Documentation index. Read this page as Markdown.
Scull
Guide

Procedure: KG and search

Build and inspect a knowledge graph, then run hybrid search.

Read as Markdown

kg_builder is a separate step from query. It reads .schema.yaml, not raw docs.

Build

scull-schemafier kg build

Reads schemas/*.schema.yaml. Only changed schema_hash entries are re-embedded. Produces:

  • kg/CURRENT, an atomic pointer to the complete published generation
  • kg/generations/<generation>/vectors.npy (one normalized float32 matrix)
  • kg/generations/<generation>/nodes.jsonl (ordered node metadata, exact embedded text, relationships)
  • kg/generations/<generation>/partitions.json (concept row ranges)
  • kg/generations/<generation>/meta.json (format/provider identity and schema catalog)
  • kg/generations/<generation>/dedup_report.json

Builds publish v2 only. Existing v1 generation or flat indexes remain readable until rebuilt. Unchanged concepts reuse vector rows when their provider, chunker version, and chunk fingerprint match; the builder still stages and validates a complete consolidated generation.

For a release build, require the complete reviewed-artifact contract:

scull validate --release
scull-schemafier kg build --release --schemas schemas --out kg

--release requires a non-empty schemaset, valid structure and attachments, current hashes, needs_review: false, and an approval fingerprint matching every artifact's current content and diagnostics. It does not perform semantic review. Ordinary builds remain available for development.

Run a build after changing the embedding model or provider (voyage → openai or local). A changed identity automatically forces re-embedding; --force also explicitly rebuilds everything:

scull-schemafier kg build --force

Failed builds leave the previous generation intact. Existing flat indexes remain readable; the next build publishes v2. Readers pin one generation per operation, and old generations remain on disk for readers already using them. Distribute CURRENT and its complete referenced generation, including normally git-ignored vectors. See the KG reference for retention details.

Deploy schemas/ and that index snapshot as one revision, then recreate or reload long-lived providers. The v2 catalog binds schema filenames, raw artifact digests, and semantic hashes before query embedding and rejects a mixed deployment. Roll back both schemas and index together; restoring only one side intentionally fails this compatibility check.

Ordinary CLI/provider startup uses the catalog without parsing every YAML file; an exact lookup loads and validates only its concept. scull validate remains the full eager schemaset check. Search memory-maps the matrix read-only. Dedup similarity uses bounded blocks. If validation reports a corrupt or unsupported index, rebuild with scull-schemafier kg build --force.

Visualize

python tools/kg_viz.py

Opens kg/graph.html — D3 force graph, self-contained. Nodes: concept (indigo), field (emerald), caveat (amber). Hover for summaries.

scull search computes:

final_score = cosine_similarity(query_vec, node_vec) + provider_lexical_weight * lexical_overlap_ratio

Lexical overlap matches significant terms (3+ chars, minus stopwords, plus operators like =>, |>). Prevents syntax queries (e.g., "pipe operator") from being buried under noise.

Search needs kg/ built. Without it, scull search reports "Run 'scull-schemafier kg build' first."