Search Today and Structured Retrieval Tomorrow

1. Status

This decision is Accepted. Site search runs on Starlight’s bundled Pagefind foundation with small presentation-adapter enhancements for Ocean tags and examples. Building structured retrieval or RAG infrastructure is explicitly deferred, not rejected.

2. Context

Readers need to find content on the published site today. Separately, atlas.knowledge-architecture §21 (“AI and RAG Readiness”) already commits Ocean-Atlas to eventually supporting retrieval-augmented generation, semantic retrieval, and other AI consumption of its knowledge, while keeping the canonical model independent of any specific AI technology.

adr.atlas-builder-boundary deliberately excludes search indexing, embeddings, RAG, a vector database, and a runtime Atlas API from Core V1’s responsibilities, calling them “future stories.” That leaves two open questions this decision answers: what powers search right now, and what constraint governs whatever powers retrieval later.

3. Decision

Today: Ocean-Atlas does not operate a separate search service. @astrojs/starlight supplies Pagefind as the static search foundation; Pagefind indexes rendered dist/ output at build time and runs client-side. The Astro/Starlight adapter deliberately enhances that foundation: OceanSearch.astro normalizes hyphenated tag queries, and example pages emit Pagefind title and weighted tag metadata so examples are discoverable by their usage vocabulary. No pagefind dependency is declared directly in builders/astro-starlight/package.json; it arrives through Starlight.

Tomorrow: When Ocean-Atlas builds structured retrieval or a RAG pipeline, it must be built on top of builders/core’s normalized AtlasModel — the same structured, typed, ID-bearing model adr.atlas-builder-boundary defines and adr.canonical-markdown-build-model requires every presentation adapter to consume — not by scraping or re-parsing the rendered HTML in dist/. builders/core already produces exactly this kind of artifact today as a build-time intermediate (builders/astro-starlight/.generated/atlas-model.json), even though it is currently adapter-local and not yet a published, stable contract for external consumers.

This constraint follows directly from the one-directional dependency rule in adr.atlas-builder-boundary: generated presentation output is a disposable projection of canonical knowledge, not a second source of truth. A retrieval system built by scraping dist/ HTML would depend on Starlight’s rendering details and would re-derive — imperfectly — the IDs, relationships, authority levels, and lifecycle status the structured model already carries.

4. Alternatives Considered

4.1 Build a dedicated search/indexing service now

Rejected for this decision. No retrieval requirement currently exceeds what static, client-side search provides, and adr.atlas-builder-boundary already scoped search indexing out of Core V1 as unproven.

4.2 Replace Pagefind with a hosted search product (e.g., Algolia)

Rejected for now. A hosted search product would add an external service dependency and a data-export step for no capability Ocean-Atlas currently needs beyond what Starlight already provides by default.

4.3 Let a future RAG pipeline scrape the rendered site

Rejected. Scraping dist/ HTML would violate the one-directional data flow in adr.atlas-builder-boundary (generated output must never become a second source of truth), would silently break whenever Starlight’s markup changed, and would have to reconstruct metadata, IDs, and relationships that already exist, intact, in the normalized model.

4.4 Design and freeze a public retrieval API now

Rejected. atlas.knowledge-architecture §21 commits to eventual retrieval readiness in principle, but no concrete consumer exists yet. Designing a frozen public contract before a real consumer proves its shape risks the same over-generalization risk adr.atlas-builder-boundary already flags for Core.

5. Consequences

Benefits

• Site search works today without a separate hosted service or runtime search infrastructure.
• The constraint on future retrieval is decided now, while the codebase is small, instead of being discovered the hard way after a scraping-based prototype has to be thrown away.
• Because builders/core already emits a structured model as a build step, “consume Core’s structured output” is not a future rewrite — it is pointing a new consumer at something that already exists.

Trade-offs

• Pagefind’s static, client-side index has no semantic ranking, no relationship-aware retrieval, and no authority- or lifecycle-awareness — it cannot deliver on the “Retrieval Readiness” goals in atlas.knowledge-architecture §21 by itself.
• .generated/atlas-model.json is currently an adapter-local intermediate file, not a documented, stable, externally consumable contract; a real future consumer will still require deliberate work to publish and version it as one.
• Deferring the decision means Ocean-Atlas has no current answer to semantic search, only a documented constraint on how that answer must eventually be built.

6. Implementation Implications

• Adapter-level Pagefind indexing and query improvements are allowed while Pagefind meets the site’s actual search needs; a separate search backend requires a new architectural decision and demonstrated requirements.
• Any future retrieval, embeddings, or RAG work is scoped as a new consumer of builders/core’s AtlasModel, not as a modification to builders/astro-starlight or a scraper over dist/.
• If .generated/atlas-model.json (or an equivalent) becomes a real external contract, it needs its own stability guarantees — schema versioning already exists (AtlasModel.schemaVersion) — and its own publishing decision at that time.

7. Validation and Compliance Rules

• A future retrieval or RAG proposal that reads dist/ HTML, Starlight’s generated content directory, or any other adapter-owned output as its primary data source does not comply with this decision.
• A future retrieval or RAG proposal that reads canonical Markdown directly, bypassing builders/core’s parsing and validation, does not comply with this decision either — normalized, validated output is the required interface, not raw source files.
• Removing or replacing Pagefind does not require revisiting this decision unless the replacement also intends to become the foundation for structured retrieval.

8. Related Knowledge

• adr.atlas-builder-boundary — scopes search, embeddings, RAG, and a runtime Atlas API out of Core V1, and establishes the one-directional data flow this decision’s “tomorrow” constraint enforces.
• adr.canonical-markdown-build-model — the structured build pipeline that produces the model future retrieval must consume.
• adr.stable-knowledge-identity — the stable IDs future retrieval results and citations depend on.
• atlas.knowledge-architecture — §21 “AI and RAG Readiness” states the longer-term retrieval goals this decision defers, not abandons.

These semantic relationships are declared in the document metadata.