# Knowledge Graph Builder

> Documentation index: [llms.min.txt](https://caseyrfsmith.github.io/scull/llms.min.txt) · Document manifest: [documents.json](https://caseyrfsmith.github.io/scull/documents.json)

Breadcrumbs: Reference > Knowledge Graph Builder

The knowledge graph is built from `.schema.yaml` files by `scull-schemafier kg build`. It creates a vector index for semantic search, structured node/edge relationships for queries like `related` and `dedup`, and a dedup report for managing near-duplicate caveats.

## Quick Start

```bash
# Build or update the KG incrementally
scull-schemafier kg build

# Force full rebuild (changes embedding model, switches providers, or corrupted index)
scull-schemafier kg build --force

# Output summary shows what was rebuilt vs skipped
rebuilt: deployment_config
skipped: query_operators, job_runtime, ...
nodes: 248
edges: 312
dedup_candidates: 67
new_dedup_candidates: 3
```

## Architecture

### Node Types

Three node kinds represent different levels of granularity:

| Kind      | ID Format               | Purpose                                                               |
| --------- | ----------------------- | --------------------------------------------------------------------- |
| `concept` | `{concept}`             | The entire schema namespace — embeds field names and caveat summaries |
| `field`   | `{concept}:{field_id}`  | A single field definition — embeds description + value type           |
| `caveat`  | `{concept}:{caveat_id}` | A behavioral caveat — embeds severity, summary, evidence              |

Node IDs are used throughout the system to reference specific pieces of information. When you call
`scull get deployment_config values`, it resolves through
`field:deployment_config:values` → load schema → return field dict.

### Edge Relationships

Three edge types connect nodes into a navigable graph:

```
concept → field     (has_field)        # "this concept owns this field"
concept → caveat    (has_caveat)       # "this concept has this caveat"
caveat → field      (applies_to)       # "this caveat affects these fields"
```

The `applies_to` edges power `scull related <concept> <field>` — look up which caveats apply to a field by traversing edges backward from field nodes.

### Text Chunks for Embeddings

Each node kind uses different text for embedding construction:

```python
# Concept node — broad context with field names and caveat summaries
f"Concept: {concept}. Fields: {fields}. Caveats: {summaries}"

# Field node — focused on behavior
f"Field {concept}.{field_name} ({value_type}): {description}"

# Caveat node — includes severity tag and raw evidence
f"Caveat [{severity}] {concept}: {summary} Evidence: {evidence}"
```

This gives each node a distinct embedding profile while preserving cross-node context (concept-level nodes include their fields/caveats, field-level nodes include descriptions, caveat nodes include severity).

## File Layout

After `kg build`:

```
kg/
├── CURRENT                 # Atomically published generation name
├── generations/
│   └── gen-<id>/            # Complete immutable snapshot
│       ├── vectors.npy
│       ├── nodes.jsonl
│       ├── partitions.json
│       ├── meta.json
│       └── dedup_report.json
└── (v1 generation and flat layouts remain readable)
```

New builds always publish format v2. Existing v1 indexes remain queryable, but rebuilding them
creates v2 rather than modifying the old files.

### Vectors, nodes, and partitions

`vectors.npy` is one unit-normalized float32 `(N, D)` matrix. `nodes.jsonl` has one ordered record per
row, including the node ID, concept, kind, exact embedded text, display fields, and caveat
`applies_to` relationships. `partitions.json` maps each concept to a contiguous `[start, stop)` row
range. Concept-filtered search can therefore slice the memory-mapped matrix directly.

`meta.json` records the format version, provider/model identity, vector dimension and dtype, and a
catalog for every schema. Catalog entries bind the filename, semantic schema hash, raw artifact
digest, review/count metadata, chunker version, and chunk fingerprint.

### Incremental reuse

The builder fingerprints each concept's ordered node IDs and exact chunk text. If the provider,
chunker version, and fingerprint match, it reuses that concept's rows while assembling a new
consolidated matrix. A raw-file metadata change can republish the catalog without an embedding API
call. A true no-op validates and retains the current generation.

If the configured build-provider identity changes, the builder forces a full re-embed without
`--force`. Search checks exact identity or an explicitly recorded compatibility family plus all
stored dimensions before making a query embedding request. The sole cross-model family currently
supported is Voyage 4 Large/4/Lite/Nano; Voyage Code 4 and unknown providers remain exact-identity
only. A mismatch returns an actionable error asking for `scull-schemafier kg build --force`;
equal-width vectors from incompatible models are rejected too.
When a provider cannot advertise its query dimension in advance, the returned vector is checked
before scoring. Empty published indexes return no results without requesting an embedding.

Builds write and validate a complete private generation before replacing `CURRENT`. Failed embedding,
validation, or publication leaves the previously active generation usable. Old generations are retained
for readers that already pinned them, so operators should account for their disk use and prune only when
long-lived readers have finished.

`--force` is also the recovery command for corrupt, unsupported, or incompatible index artifacts.

### Dedup Report

Post-build analysis finds within-concept caveat pairs whose cosine similarity exceeds `EMBEDDING_DEDUP_THRESHOLD = 0.90`. These are potential duplicates the text-based reconciler missed because they cover the same behavior but use different phrasing.

**Why this matters:** Two caveats like "pipe operator returns first arg" and "first argument of pipe operator is returned unchanged" cover the same semantic content but would be flagged as distinct during reconciliation if only text-based dedup (SequenceMatcher on summaries) ran. Embedding similarity catches these cross-phrasing cases.

**Only flags same-concept pairs.** Different concepts having similar caveats is normal and not reported.
The comparison runs in bounded 512-row blocks instead of allocating a full all-pairs matrix.

## Algorithm Details

### Build Pipeline

```
schemas/*.schema.yaml
    │
    ▼ (load YAML)
Catalog/fingerprint check ←── schema + chunk + provider identity
    ▼
Rebuilt: concepts that changed or need model rebuild
Skipped: concepts already embedded (hash match)
    │
    ▼ (_concept_chunks for each rebuilt concept)
Chunk texts: concept + field + caveat representations
    │
    ▼ (provider.embed())
Embedding/reused vectors → vectors.npy
Ordered records/ranges → nodes.jsonl + partitions.json + meta.json
    │
    ▼ (_find_embedding_duplicates)
Cosine similarity matrix → dedup_report.json
```

#### Incremental Rebuild Logic

```python
if provider_and_chunk_fingerprint_match and not force:
    skipped.append(concept)      # Reuse the old partition rows
else:
    rebuilt.append(concept)      # Re-embed this concept
```

### Cosine Similarity Search

Hybrid scoring combines two signals:

```
final_score = cosine_similarity(query_vec, node_vec)
            + LEXICAL_WEIGHT * lexical_overlap_ratio
```

Where:

- `lexical_weight = 0.3` for Voyage/OpenAI or `0.1` for local MiniLM — deterministic term matching bonus
- `lexical_overlap_ratio = matching_terms / total_query_terms`

**Cosine component:** Pure vector-space proximity. Handles fuzzy/synonym matches ("zero decimal currencies" ≈ "currency codes without pennies").

**Lexical component:** Deterministic substring matching on significant terms (3+ chars, minus stopwords, plus operators like `=>`, `|>`, `??`). Prevents short niche queries like "=>" from being buried under noise.

**Max achievable lexical overlap:** 1.0 (all query terms matched exactly). The combined maximum is
1.3 for hosted providers and 1.1 for local MiniLM. Ranking uses that full score; the public display
score is capped at 1.0.

#### Term Extraction

Significant terms are extracted from both the query and node text using `_significant_terms()`:

```python
# Extracts: words 3+ chars (minus stopwords) + punctuation clusters
query = "how do I make config immutable?"
terms = ["how", "make", "config", "immutable"]  # "do", "i", "?" dropped

node_text = "Immutable flag prevents updates to deployed configuration"
matched = ["config", "immutable"]
lexical_bonus = 2 / 4 × 0.3 = 0.15
```

This specifically targets domain-specific identifiers (operator syntax, config keys, method names) that embeddings alone under-rank.

#### Degraded lexical fallback

If query embedding fails after the index has passed its provider-identity and dimension checks,
`SearchEngine` ranks only nodes with lexical matches. Each returned result includes
`degraded: true`, `degraded_reason: "query_embedding_failed"`, and `cosine_score: null`. With no
lexical matches it returns no results, avoiding arbitrary zero-signal rankings. Index metadata
incompatibilities still fail closed and never enter degraded mode.

### Dedup Detection

Runs after embeddings are generated. For each concept, computes cosine similarity across caveat
embeddings in bounded blocks:

```python
# Get all caveat embeddings
caveat_embs = embeddings[:, :][mask_for_caveat_nodes]

# Normalize vectors
normed = caveat_embs / np.linalg.norm(caveat_embs, axis=1, keepdims=True)

# Each concept is processed in bounded row blocks.
for row_block, column_block in caveat_partition_blocks:
    similarities = row_block @ column_block.T
    yield_pairs_above(0.90, similarities)
```

Only reports pairs exceeding the threshold. Sorted by descending similarity so reviewers see most-suspicious pairs first. New pairs detected compared to prior `dedup_report.json` (stale duplicates aren't re-reported on every rebuild).

### Model Swap Safety

Changing the configured provider or model triggers an automatic full rebuild:

```python
# Before your change: meta["embedding"]["identity"] = "VoyageEmbed:voyage-4-lite"
# You change to:       provider.identity() = "OpenAIEmbed:text-embedding-3-small"

# Next run detects mismatch:
published_model = meta["embedding"]["identity"]
current_model = provider.identity()

if published_model != current_model:
    force = True  # Never silently compare mismatched vectors
```

Vectors from incompatible models live in completely different embedding spaces. Comparing a `voyage-4-lite` vector against a `text-embedding-3-small` vector would produce high-similarity garbage — no error message, just wrong answers forever. The auto-force guard exists solely for this reason.

## Environment Variables

No direct env vars control `kg_builder.py` — it reads:

- `SCULL_EMBED_PROVIDER` — from `get_provider()` (defaults to `"voyage"`)
- `VOYAGE_API_KEY` or `OPENAI_API_KEY` — depending on provider selection
- Schema path resolved from CLI args or defaults (`schemas/`)
- KG output path resolved from CLI args or defaults (`kg/`)

To override defaults, pass explicit paths:

```bash
scull-schemafier kg build --schemas /custom/schemas/ --out /custom/kg/
```

## Python API

For programmatic access (used internally by `FileSchemaProvider.search()`):

```python
from pathlib import Path
from scull.query import FileSchemaProvider, SearchEngine
from scull.schemafier.kg_builder import build

# Build/update index
result = build(
    schema_dir=Path("./schemas"),
    kg_dir=Path("./kg"),
    force=False  # Set True for full rebuild
)

print(result["rebuilt"])          # list of concept names
print(result["skipped"])          # list of concept names
print(result["nodes"], result["edges"])
print(result["dedup_candidates"]) # within-concept dup caveat pairs
print(result["new_dedup_candidates"]) # newly surfaced this run

# Run search through the shared query layer
schema_provider = FileSchemaProvider(Path("./schemas"), kg_dir=Path("./kg"))
results = schema_provider.search("make config immutable", top_k=5, concepts=None)

# Returns List[Dict]:
# [{"node_id": "field:deployment_config:immutable", "score": 1.0, ...}, ...]
```

`SearchEngine` is the implementation used by `FileSchemaProvider.search()`, `scull search`, the
schemafier search command, and the local server.

The current engine performs brute-force NumPy scoring over selected rows. It is intended for
schemasets with thousands to tens of thousands of nodes; there is no approximate-nearest-neighbor
index or automatic generation garbage collection. Old generations remain until an operator removes
ones no active long-lived reader can still reference.

A reproducible synthetic run at 1,000 schemas / 51,000 nodes measured 265.4 ms median provider
startup with zero YAML parses, 313.9 ms first full search, 12.582 ms median warm local scoring, and
0.024 ms for one concept on the test machine. The generation occupied 214.645 MiB, and even a no-op
or one-concept incremental publication took roughly eight seconds because builds still validate and
assemble the complete generation. Hosted query-embedding latency and clean-process server memory
remain separate measurements.