Knowledge Chunking and Indexing Pipeline

1. Status

This decision is Accepted. It resolves the two questions adr.vector-storage-and-embedding-provider explicitly deferred: chunking strategy, and embedding refresh/versioning strategy.

2. Context

Turning canonical Ocean-Atlas knowledge into embeddings requires deciding how documents are split into retrieval-sized units, how those units keep an identity that survives reorganization, and how the resulting index stays correct as content changes — without re-doing all of that work on every process start, and without corrupting the index if more than one process tries to update it at once.

atlas.knowledge-architecture §21.3 (“Chunking Readiness”) already sets the governing principle: chunk boundaries should follow logical knowledge boundaries, not arbitrary character or token limits, and chunks should preserve stable identity, section boundaries, metadata, relationships, and source attribution. adr.stable-knowledge-identity separately establishes that a document’s stable ID — not its file path or rendered route — is the only thing relationships and cross-references may depend on, because paths and routes change under reorganization and IDs don’t. Chunk identity needs the same property.

adr.vector-storage-and-embedding-provider already established that everything in the ocean_atlas Postgres database is a rebuildable cache, never authoritative, and that indexing must not become a source of truth. This ADR is the concrete mechanism that keeps that cache correct over time.

3. Decision

3.1 Chunking algorithm

One chunking function handles both documents and examples — not two separate code paths:

1. Split content on ## (level-2 heading) boundaries — this repository’s own established convention for every canonical document.
2. If a resulting section still exceeds a size budget (~1500 characters), recursively split on ### within it; if still oversized, fall back to paragraph boundaries.
3. Examples default to one chunk (title + summary + knowledge.content, not sub-split, since examples are normally short and already a natural single retrieval unit) — but an example that exceeds the same size budget goes through the same recursive splitting as a document, rather than producing one oversized chunk. This guards against violating the embedding model’s input limits regardless of how large a future example description gets.

3.2 Chunk identity

A chunk’s ID is derived from the document’s stable ID, never its route or file path:

<document-stable-id>#<heading-slug>
<document-stable-id>#<heading-slug>/<sub-heading-slug>   (when a section is further split)

For example: adr.knowledge-api-service#3-decision. This ID is stable exactly as long as the heading text producing it doesn’t change — a renamed heading produces a new chunk ID (indexed as new) and orphans the old one (removed by the reconciliation diff in 3.4), never a route or file path, which are presentation/location details adr.stable-knowledge-identity deliberately keeps out of identity.

3.3 Index versioning

Two columns govern whether a chunk needs re-embedding, not just one:

• content_hash — sha256 of the chunk’s text (same convention builders/core/src/examples.ts already uses for example files), detects a content change.
• index_version — a string encoding the chunking-algorithm version, embedding provider, model, and dimensions (e.g. 1:voyage:voyage-4:1024), detects a recipe change even when the underlying text hasn’t changed at all.

A chunk is re-embedded when its stored content_hash differs from a fresh computation, or its stored index_version differs from the current one. This is what makes a future voyage-5 upgrade, a chunking-algorithm change, or a dimension change trigger correct, complete re-embedding — a content hash alone cannot detect any of those.

3.4 Reconciliation and deletion

Each indexing pass computes the full set of chunk IDs the current AtlasModel should produce, then:

• embeds and upserts any ID that’s new, or whose content_hash/index_version no longer matches;
• deletes every stored chunk row whose ID is not in that current expected set.

This one rule handles heading renames, section removals, and whole-document deletions uniformly — no special-casing “document deleted” versus “section deleted” — and is what prevents removed knowledge from remaining searchable forever.

3.5 Concurrency safety

Before an indexing pass runs, the process attempts pg_try_advisory_lock(<fixed key>) on a dedicated, held-open connection. If the lock isn’t acquired, this process skips its own indexing attempt entirely and simply serves semantic search against whatever is already in the database — it does not block, retry immediately, or duplicate work. The lock is released explicitly when indexing finishes, and automatically by Postgres if the holding connection drops (process crash or restart), so a crashed indexer can never leave the lock permanently stuck. This is what keeps concurrent indexing safe once ocean-atlas runs more than one Fly Machine, without needing a distributed job scheduler.

3.6 Availability semantics

A singleton knowledge_index_meta row tracks last_completed_at and completed_index_version for the most recent fully completed reconciliation pass (completion defined by 3.7). Two distinct states follow from this, and the difference matters for health reporting:

• Usable — last_completed_at is not null: at least one reconciliation has ever fully completed. Semantic search is available, even if that completed generation reflects an older index_version or slightly older content than what’s canonical right now.
• Current — completed_index_version equals the code’s present index-version constant and no reconciliation is currently mid-flight with a newer diff pending: the served index matches both the current recipe and current content.

Semantic search is available whenever the index is usable, regardless of whether a new background reconciliation is currently in progress — it does not need to be current. It is unavailable (503) only when no index generation has ever completed at all (the very first deploy, before indexing has finished once).

This means a routine restart with an already-good index in Postgres does not make semantic search unavailable while that restart’s reconciliation catches up on the day’s changes, or while a model/algorithm upgrade’s re-embedding is still in flight — the service keeps answering from the last-known-good (usable, possibly not yet current) index while refreshing in the background:

Server starts
    │
    ├── Existing Atlas routes available immediately
    │
    ├── Check existing index state
    │      ├── usable (last_completed_at set) → semantic search available
    │      └── absent (never completed)        → semantic search 503
    │
    └── Background index reconciliation (runs regardless, independently)
           │
           ├── acquire indexing lock (pg_try_advisory_lock)
           ├── chunk canonical documents and examples
           ├── compare ID + content_hash + index_version
           ├── determine diff (new / changed / removed)
           ├── call the embedding provider for every new/changed chunk
           ├── (all external work succeeded)
           ├── BEGIN — upsert, delete, update knowledge_index_meta — COMMIT
           └── index is now current (see 3.7)

3.7 Atomic publication

A reconciliation pass must not partially publish a new index. Chunking, diff computation, and every required embedding-provider call complete first, against data still in memory — nothing in knowledge_chunks or knowledge_index_meta is touched yet. Only once all of that external work has succeeded does persistence happen, and it happens as one PostgreSQL transaction: the upserts, the deletions, and the knowledge_index_meta update (last_completed_at, completed_index_version) all commit together or none of them do.

If chunking or an embedding-provider call fails, nothing has been written and the previously completed index — still fully intact — remains exactly as usable as it was before this pass started. If the transaction itself fails partway, Postgres rolls it back for the same reason. Either way, “usable” (3.6) always means a complete, self-consistent generation, never a half-applied one — an index that is genuinely last-known-good, not merely last-attempted.

4. Alternatives Considered

4.1 Full re-embedding on every process start

Rejected. Mirrors how buildAtlasModel() runs today, but embeddings cost real money and real latency per chunk — paying that on every deploy and every Fly scale-to-zero wake-up is avoidable waste once a versioned, hash-gated diff can do the same job proportional to what actually changed.

4.2 Blocking server startup until indexing completes

Rejected. Would make every restart briefly unavailable for semantic search even when a perfectly good index already exists in Postgres from before — exactly the failure mode 3.6 avoids. Existing keyword/document routes have no dependency on indexing at all and must never wait on it either.

4.3 No concurrency protection

Rejected once ocean-atlas might run more than one Machine: without a lock, multiple replicas would redundantly chunk, embed, and write the same rows, wasting Voyage API calls and risking write races on the same table.

4.4 Chunk IDs derived from route or file path

Rejected for the same reason adr.stable-knowledge-identity rejected it for documents: routes and paths change when content is reorganized, and a chunk ID that changed on every reorganization would orphan and re-embed unrelated content for no real reason.

4.5 Unbounded chunk size for examples

Rejected. “One chunk per example” is the right default, but an unbounded exception would eventually violate the embedding model’s input limits for an unusually large example description.

4.6 Publishing each chunk as it’s embedded, without a wrapping transaction

Rejected. Writing upserts and deletions incrementally as embedding calls complete, rather than atomically at the end, means a failure partway through (a Voyage request error, a dropped Postgres connection) leaves knowledge_chunks in a half-updated state — neither the old generation nor the new one, and worse than either. That directly breaks the “usable index” guarantee 3.6 depends on: usable must mean a complete, self-consistent generation, never a partial one. 3.7 exists specifically to rule this out.

5. Consequences

Benefits

• Restarts and routine redeploys don’t punish semantic-search availability when a good index already exists.
• Safe under multiple replicas without a distributed job scheduler.
• Correct under deletion: removed knowledge stops being searchable instead of lingering indefinitely.
• A future embedding-model or chunking-algorithm change is a versioned, self-healing re-index, not a manual cache-clear step.
• A failed reconciliation (Voyage error, Postgres error, process crash) never corrupts the served index — the previous completed generation stays fully intact and searchable throughout, by construction rather than by luck.

Trade-offs

• Two tables instead of one (knowledge_chunks plus the knowledge_index_meta singleton), a small amount of extra schema versus the simplest possible design.
• The advisory-lock mechanism is one more moving part to reason about, even though it stays inactive (uncontended) until a second replica actually exists.
• A stale-but-previously-completed index can be served for a while during a slow background reconciliation — accepted deliberately, since eventual consistency here is strictly better than an availability outage over the same window. During a model/algorithm upgrade specifically, this means semantic search briefly serves results embedded under the previous index_version rather than the current one — usable, not yet current (3.6) — until that pass’s transaction commits.
• All embeddings for a pass must be held in memory until the commit; for this corpus’s size that’s inconsequential, but it means a single pass isn’t itself incrementally streamed to disk as it progresses.

6. Implementation Implications

• knowledge_chunks carries content_hash and index_version alongside the embedding; knowledge_index_meta is a singleton row carrying last_completed_at and completed_index_version.
• services/knowledge-api’s indexing code must attempt the advisory lock before doing any chunking or embedding work, must complete every required embedding-provider call before touching the database at all, and must apply the resulting upserts, deletions, and knowledge_index_meta update inside one transaction (3.7) — never incrementally.
• GET /v1/health exposes whether semantic search is usable (last_completed_at set) and, separately, whether the served index is current (completed_index_version equals the code’s present index-version constant) — two different facts, not one boolean.
• GET /v1/semantic-search returns 503 only in the “never completed” (not usable) state, never merely because a refresh is in progress or the usable index isn’t yet current.

7. Validation and Compliance Rules

• A chunk row belonging to a previous index_version is valid evidence of a usable index — semantic search may serve it — but not a current one; health and monitoring must derive “current” from knowledge_index_meta.completed_index_version against the code’s present constant, not by inspecting individual chunk rows.
• A reconciliation pass must not mutate knowledge_chunks or knowledge_index_meta until every required embedding-provider call for that pass has already succeeded, and must then apply all resulting writes in a single transaction — an implementation that upserts or deletes incrementally as it goes does not comply with 3.7.
• Every indexing pass must compute deletions from the full current expected-ID set, not only additions/updates — an implementation that only upserts and never deletes does not comply.
• Indexing must not proceed without first attempting the advisory lock described in 3.5.
• A chunk ID must be derived from a document’s stable ID, never its route or sourcePath.

8. Related Knowledge

• adr.vector-storage-and-embedding-provider — the storage and embedding-provider decision this ADR completes by resolving the strategy questions it deferred.
• adr.knowledge-api-service — the service this indexing pipeline runs inside, and whose existing routes must stay unaffected by indexing failures or delays.
• adr.stable-knowledge-identity — the identity principle chunk IDs inherit directly.
• atlas.knowledge-architecture — §21.3 (“Chunking Readiness”), the governing principle behind the chunking algorithm in this decision.

These semantic relationships are declared in the document metadata.