Knowledge Graph Builder

The knowledge graph is built from .schema.yaml files by scull-schemafier kg build. It creates a vector index for semantic search, structured node/edge relationships for queries like related and dedup, and a dedup report for managing near-duplicate caveats.

Quick Start

# Build or update the KG incrementally
scull-schemafier kg build

# Force full rebuild (changes embedding model, switches providers, or corrupted index)
scull-schemafier kg build --force

# Output summary shows what was rebuilt vs skipped
rebuilt: deployment_config
skipped: query_operators, job_runtime, ...
nodes: 248
edges: 312
dedup_candidates: 67
new_dedup_candidates: 3

Architecture

Node Types

Three node kinds represent different levels of granularity:

Node IDs are used throughout the system to reference specific pieces of information. When you call scull get deployment_config values, it resolves through field:deployment_config:values → load schema → return field dict.

Edge Relationships

Three edge types connect nodes into a navigable graph:

concept → field     (has_field)        # "this concept owns this field"
concept → caveat    (has_caveat)       # "this concept has this caveat"
caveat → field      (applies_to)       # "this caveat affects these fields"

The applies_to edges power scull related <concept> <field> — look up which caveats apply to a field by traversing edges backward from field nodes.

Text Chunks for Embeddings

Each node kind uses different text for embedding construction:

# Concept node — broad context with field names and caveat summaries
f"Concept: {concept}. Fields: {fields}. Caveats: {summaries}"

# Field node — focused on behavior
f"Field {concept}.{field_name} ({value_type}): {description}"

# Caveat node — includes severity tag and raw evidence
f"Caveat [{severity}] {concept}: {summary} Evidence: {evidence}"

This gives each node a distinct embedding profile while preserving cross-node context (concept-level nodes include their fields/caveats, field-level nodes include descriptions, caveat nodes include severity).

File Layout

After kg build:

kg/
├── CURRENT                 # Atomically published generation name
├── generations/
│   └── gen-<id>/            # Complete immutable snapshot
│       ├── vectors.npy
│       ├── nodes.jsonl
│       ├── partitions.json
│       ├── meta.json
│       └── dedup_report.json
└── (v1 generation and flat layouts remain readable)

New builds always publish format v2. Existing v1 indexes remain queryable, but rebuilding them creates v2 rather than modifying the old files.

Vectors, nodes, and partitions

vectors.npy is one unit-normalized float32 (N, D) matrix. nodes.jsonl has one ordered record per row, including the node ID, concept, kind, exact embedded text, display fields, and caveat applies_to relationships. partitions.json maps each concept to a contiguous [start, stop) row range. Concept-filtered search can therefore slice the memory-mapped matrix directly.

meta.json records the format version, provider/model identity, vector dimension and dtype, and a catalog for every schema. Catalog entries bind the filename, semantic schema hash, raw artifact digest, review/count metadata, chunker version, and chunk fingerprint.

Incremental reuse

The builder fingerprints each concept's ordered node IDs and exact chunk text. If the provider, chunker version, and fingerprint match, it reuses that concept's rows while assembling a new consolidated matrix. A raw-file metadata change can republish the catalog without an embedding API call. A true no-op validates and retains the current generation.

If the configured build-provider identity changes, the builder forces a full re-embed without --force. Search checks exact identity or an explicitly recorded compatibility family plus all stored dimensions before making a query embedding request. The sole cross-model family currently supported is Voyage 4 Large/4/Lite/Nano; Voyage Code 4 and unknown providers remain exact-identity only. A mismatch returns an actionable error asking for scull-schemafier kg build --force; equal-width vectors from incompatible models are rejected too. When a provider cannot advertise its query dimension in advance, the returned vector is checked before scoring. Empty published indexes return no results without requesting an embedding.

Builds write and validate a complete private generation before replacing CURRENT. Failed embedding, validation, or publication leaves the previously active generation usable. Old generations are retained for readers that already pinned them, so operators should account for their disk use and prune only when long-lived readers have finished.

--force is also the recovery command for corrupt, unsupported, or incompatible index artifacts.

Dedup Report

Post-build analysis finds within-concept caveat pairs whose cosine similarity exceeds EMBEDDING_DEDUP_THRESHOLD = 0.90. These are potential duplicates the text-based reconciler missed because they cover the same behavior but use different phrasing.

Why this matters: Two caveats like "pipe operator returns first arg" and "first argument of pipe operator is returned unchanged" cover the same semantic content but would be flagged as distinct during reconciliation if only text-based dedup (SequenceMatcher on summaries) ran. Embedding similarity catches these cross-phrasing cases.

Only flags same-concept pairs. Different concepts having similar caveats is normal and not reported. The comparison runs in bounded 512-row blocks instead of allocating a full all-pairs matrix.

Algorithm Details

Build Pipeline

schemas/*.schema.yaml
    │
    ▼ (load YAML)
Catalog/fingerprint check ←── schema + chunk + provider identity
    ▼
Rebuilt: concepts that changed or need model rebuild
Skipped: concepts already embedded (hash match)
    │
    ▼ (_concept_chunks for each rebuilt concept)
Chunk texts: concept + field + caveat representations
    │
    ▼ (provider.embed())
Embedding/reused vectors → vectors.npy
Ordered records/ranges → nodes.jsonl + partitions.json + meta.json
    │
    ▼ (_find_embedding_duplicates)
Cosine similarity matrix → dedup_report.json

Incremental Rebuild Logic

if provider_and_chunk_fingerprint_match and not force:
    skipped.append(concept)      # Reuse the old partition rows
else:
    rebuilt.append(concept)      # Re-embed this concept

Cosine Similarity Search

Hybrid scoring combines two signals:

final_score = cosine_similarity(query_vec, node_vec)
            + LEXICAL_WEIGHT * lexical_overlap_ratio

Where:

• lexical_weight = 0.3 for Voyage/OpenAI or 0.1 for local MiniLM — deterministic term matching bonus
• lexical_overlap_ratio = matching_terms / total_query_terms

Cosine component: Pure vector-space proximity. Handles fuzzy/synonym matches ("zero decimal currencies" ≈ "currency codes without pennies").

Lexical component: Deterministic substring matching on significant terms (3+ chars, minus stopwords, plus operators like =>, |>, ??). Prevents short niche queries like "=>" from being buried under noise.

Max achievable lexical overlap: 1.0 (all query terms matched exactly). The combined maximum is 1.3 for hosted providers and 1.1 for local MiniLM. Ranking uses that full score; the public display score is capped at 1.0.

Term Extraction

Significant terms are extracted from both the query and node text using _significant_terms():

# Extracts: words 3+ chars (minus stopwords) + punctuation clusters
query = "how do I make config immutable?"
terms = ["how", "make", "config", "immutable"]  # "do", "i", "?" dropped

node_text = "Immutable flag prevents updates to deployed configuration"
matched = ["config", "immutable"]
lexical_bonus = 2 / 4 × 0.3 = 0.15

This specifically targets domain-specific identifiers (operator syntax, config keys, method names) that embeddings alone under-rank.

Degraded lexical fallback

If query embedding fails after the index has passed its provider-identity and dimension checks, SearchEngine ranks only nodes with lexical matches. Each returned result includes degraded: true, degraded_reason: "query_embedding_failed", and cosine_score: null. With no lexical matches it returns no results, avoiding arbitrary zero-signal rankings. Index metadata incompatibilities still fail closed and never enter degraded mode.

Dedup Detection

Runs after embeddings are generated. For each concept, computes cosine similarity across caveat embeddings in bounded blocks:

# Get all caveat embeddings
caveat_embs = embeddings[:, :][mask_for_caveat_nodes]

# Normalize vectors
normed = caveat_embs / np.linalg.norm(caveat_embs, axis=1, keepdims=True)

# Each concept is processed in bounded row blocks.
for row_block, column_block in caveat_partition_blocks:
    similarities = row_block @ column_block.T
    yield_pairs_above(0.90, similarities)

Only reports pairs exceeding the threshold. Sorted by descending similarity so reviewers see most-suspicious pairs first. New pairs detected compared to prior dedup_report.json (stale duplicates aren't re-reported on every rebuild).

Model Swap Safety

Changing the configured provider or model triggers an automatic full rebuild:

# Before your change: meta["embedding"]["identity"] = "VoyageEmbed:voyage-4-lite"
# You change to:       provider.identity() = "OpenAIEmbed:text-embedding-3-small"

# Next run detects mismatch:
published_model = meta["embedding"]["identity"]
current_model = provider.identity()

if published_model != current_model:
    force = True  # Never silently compare mismatched vectors

Vectors from incompatible models live in completely different embedding spaces. Comparing a voyage-4-lite vector against a text-embedding-3-small vector would produce high-similarity garbage — no error message, just wrong answers forever. The auto-force guard exists solely for this reason.

Environment Variables

No direct env vars control kg_builder.py — it reads:

• SCULL_EMBED_PROVIDER — from get_provider() (defaults to "voyage")
• VOYAGE_API_KEY or OPENAI_API_KEY — depending on provider selection
• Schema path resolved from CLI args or defaults (schemas/)
• KG output path resolved from CLI args or defaults (kg/)

To override defaults, pass explicit paths:

scull-schemafier kg build --schemas /custom/schemas/ --out /custom/kg/

Python API

For programmatic access (used internally by FileSchemaProvider.search()):

from pathlib import Path
from scull.query import FileSchemaProvider, SearchEngine
from scull.schemafier.kg_builder import build

# Build/update index
result = build(
    schema_dir=Path("./schemas"),
    kg_dir=Path("./kg"),
    force=False  # Set True for full rebuild
)

print(result["rebuilt"])          # list of concept names
print(result["skipped"])          # list of concept names
print(result["nodes"], result["edges"])
print(result["dedup_candidates"]) # within-concept dup caveat pairs
print(result["new_dedup_candidates"]) # newly surfaced this run

# Run search through the shared query layer
schema_provider = FileSchemaProvider(Path("./schemas"), kg_dir=Path("./kg"))
results = schema_provider.search("make config immutable", top_k=5, concepts=None)

# Returns List[Dict]:
# [{"node_id": "field:deployment_config:immutable", "score": 1.0, ...}, ...]

SearchEngine is the implementation used by FileSchemaProvider.search(), scull search, the schemafier search command, and the local server.

The current engine performs brute-force NumPy scoring over selected rows. It is intended for schemasets with thousands to tens of thousands of nodes; there is no approximate-nearest-neighbor index or automatic generation garbage collection. Old generations remain until an operator removes ones no active long-lived reader can still reference.

A reproducible synthetic run at 1,000 schemas / 51,000 nodes measured 265.4 ms median provider startup with zero YAML parses, 313.9 ms first full search, 12.582 ms median warm local scoring, and 0.024 ms for one concept on the test machine. The generation occupied 214.645 MiB, and even a no-op or one-concept incremental publication took roughly eight seconds because builds still validate and assemble the complete generation. Hosted query-embedding latency and clean-process server memory remain separate measurements.