docs(adr): ADR-0021: Search Index Format Evolution

# ADR-0021: Search Index Format Evolution

## Status

Accepted

## Context

mdbook-htmx generates a search index (`search-index.json`) that enables full-text search. As the project evolves, the search index format will need to change to support new features:

- Additional metadata (headings, code blocks, tags)
- Scope-aware filtering
- Multi-language support
- Performance optimizations
- Integration with different search backends (lunr.js, Meilisearch, Algolia)

Changes must be backward-compatible or provide clear migration paths.

## Decision Drivers

1. **Backward Compatibility** - Older readers should gracefully handle new formats
2. **Forward Compatibility** - Newer readers should handle older formats
3. **Performance** - Index size and generation time matter
4. **Flexibility** - Support multiple search backends
5. **Clarity** - Format should be self-documenting

## Decision

**Use semantic versioning with a version field, optional fields, and a migration strategy.**

### Version Schema

```json
{
"$schema": "https://schemas.arusty.dev/mdbook-htmx/search-index/1.0.0.json",
"version": "1.0.0",
"generated": "2026-01-04T12:00:00Z",
"generator": "mdbook-htmx 0.1.0",
"config": {
"language": "en",
"minChars": 2,
"stopWords": true
},
"pages": [...]
}
```

### Version Numbering

```
MAJOR.MINOR.PATCH

MAJOR: Breaking changes (field removed, meaning changed)
MINOR: New optional fields, new page fields
PATCH: Bug fixes, documentation
```

### Compatibility Rules

| Reader Version | Index Version | Behavior |
|----------------|---------------|----------|
| 1.x | 1.x | Full compatibility |
| 1.x | 2.x | Error or degraded mode |
| 2.x | 1.x | Migration applied |

### Version Detection

```javascript
function loadSearchIndex(index) {
const version = index.version || "0.0.0";
const [major] = version.split(".").map(Number);

switch (major) {
case 0:
return migrateV0(index);
case 1:
return index; // Current version
case 2:
throw new Error(`Search index v${version} requires mdbook-htmx upgrade`);
default:
throw new Error(`Unknown search index version: ${version}`);
}
}
```

### Format Evolution Plan

#### Version 1.0.0 (Current)

Base format with core fields:

```json
{
"version": "1.0.0",
"generated": "2026-01-04T12:00:00Z",
"config": {
"language": "en"
},
"pages": [
{
"path": "/guide/intro.html",
"title": "Introduction",
"content": "Full text content for search...",
"headings": ["Getting Started", "Prerequisites"]
}
]
}
```

#### Version 1.1.0 (Planned)

Add scope and tags (optional fields):

```json
{
"version": "1.1.0",
"pages": [
{
"path": "/internal/roadmap.html",
"title": "Product Roadmap",
"content": "...",
"headings": ["Q1 Goals", "Q2 Goals"],
"scope": "internal", // NEW
"tags": ["planning"] // NEW
}
]
}
```

#### Version 1.2.0 (Planned)

Add code block indexing:

```json
{
"version": "1.2.0",
"pages": [
{
"path": "/api/auth.html",
"title": "Authentication API",
"content": "...",
"headings": [...],
"codeBlocks": [ // NEW
{
"language": "rust",
"content": "fn authenticate(...)"
}
]
}
]
}
```

#### Version 1.3.0 (Planned)

Add heading anchors for deep linking:

```json
{
"version": "1.3.0",
"pages": [
{
"path": "/guide/config.html",
"title": "Configuration",
"content": "...",
"sections": [ // NEW (replaces headings)
{
"id": "basic-setup",
"title": "Basic Setup",
"level": 2,
"content": "..."
}
]
}
]
}
```

#### Version 2.0.0 (Future)

Breaking changes if needed:

```json
{
"version": "2.0.0",
"format": "segmented", // NEW format type
"segments": {
"pages": [...],
"headings": [...], // Separate index
"code": [...] // Separate index
}
}
```

### Migration Strategies

#### V0 to V1 Migration

```javascript
function migrateV0(index) {
// V0 had no version field
return {
version: "1.0.0",
generated: new Date().toISOString(),
config: { language: "en" },
pages: (index.docs || index.pages || []).map(doc => ({
path: doc.url || doc.path,
title: doc.title,
content: doc.body || doc.content,
headings: doc.breadcrumbs || []
}))
};
}
```

#### Field Presence Handling

```javascript
function getSearchableContent(page) {
const parts = [page.title, page.content];

// Handle optional fields gracefully
if (page.headings) {
parts.push(...page.headings);
}

if (page.sections) {
parts.push(...page.sections.map(s => s.title));
parts.push(...page.sections.map(s => s.content));
}

if (page.codeBlocks) {
parts.push(...page.codeBlocks.map(b => b.content));
}

if (page.tags) {
parts.push(...page.tags);
}

return parts.filter(Boolean).join(' ');
}
```

### Backend-Specific Formats

#### Lunr.js (Client-Side)

```json
{
"version": "1.0.0",
"backend": "lunr",
"index": {
"version": "2.3.9",
"fields": ["title", "content", "headings"],
"fieldVectors": [...],
"invertedIndex": [...]
},
"store": {
"/guide/intro.html": {
"title": "Introduction",
"headings": ["Getting Started"]
}
}
}
```

#### Meilisearch

```json
{
"version": "1.0.0",
"backend": "meilisearch",
"documents": [
{
"id": "guide-intro",
"path": "/guide/intro.html",
"title": "Introduction",
"content": "...",
"headings": ["Getting Started"],
"_tags": ["guide"]
}
],
"settings": {
"searchableAttributes": ["title", "headings", "content"],
"filterableAttributes": ["_tags", "scope"]
}
}
```

#### Algolia

```json
{
"version": "1.0.0",
"backend": "algolia",
"records": [
{
"objectID": "guide-intro",
"path": "/guide/intro.html",
"title": "Introduction",
"content": "...",
"hierarchy": {
"lvl0": "Guide",
"lvl1": "Introduction",
"lvl2": "Getting Started"
}
}
]
}
```

### Schema Validation

JSON Schema for validation:

```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://schemas.arusty.dev/mdbook-htmx/search-index/1.0.0.json",
"title": "mdbook-htmx Search Index",
"type": "object",
"required": ["version", "pages"],
"properties": {
"version": {
"type": "string",
"pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+$"
},
"generated": {
"type": "string",
"format": "date-time"
},
"generator": {
"type": "string"
},
"config": {
"$ref": "#/$defs/config"
},
"pages": {
"type": "array",
"items": { "$ref": "#/$defs/page" }
}
},
"$defs": {
"config": {
"type": "object",
"properties": {
"language": { "type": "string", "default": "en" },
"minChars": { "type": "integer", "default": 2 },
"stopWords": { "type": "boolean", "default": true }
}
},
"page": {
"type": "object",
"required": ["path", "title", "content"],
"properties": {
"path": { "type": "string" },
"title": { "type": "string" },
"content": { "type": "string" },
"headings": {
"type": "array",
"items": { "type": "string" }
},
"scope": { "type": "string" },
"tags": {
"type": "array",
"items": { "type": "string" }
},
"sections": {
"type": "array",
"items": { "$ref": "#/$defs/section" }
},
"codeBlocks": {
"type": "array",
"items": { "$ref": "#/$defs/codeBlock" }
}
}
},
"section": {
"type": "object",
"required": ["id", "title", "level"],
"properties": {
"id": { "type": "string" },
"title": { "type": "string" },
"level": { "type": "integer", "minimum": 1, "maximum": 6 },
"content": { "type": "string" }
}
},
"codeBlock": {
"type": "object",
"required": ["content"],
"properties": {
"language": { "type": "string" },
"content": { "type": "string" },
"filename": { "type": "string" }
}
}
}
}
```

### Implementation

#### Rust Index Generator

```rust
// src/search/index.rs
use serde::{Serialize, Deserialize};
use semver::Version;

pub const CURRENT_VERSION: &str = "1.0.0";

#[derive(Serialize, Deserialize)]
pub struct SearchIndex {
pub version: String,
pub generated: String,
pub generator: String,
pub config: SearchConfig,
pub pages: Vec,
}

impl SearchIndex {
pub fn new(config: SearchConfig) -> Self {
Self {
version: CURRENT_VERSION.to_string(),
generated: chrono::Utc::now().to_rfc3339(),
generator: format!("mdbook-htmx {}", env!("CARGO_PKG_VERSION")),
config,
pages: vec![],
}
}

pub fn is_compatible(&self, reader_version: &str) -> bool {
let index_ver = Version::parse(&self.version).unwrap();
let reader_ver = Version::parse(reader_version).unwrap();

// Same major version is compatible
index_ver.major == reader_ver.major
}
}

#[derive(Serialize, Deserialize)]
pub struct SearchPage {
pub path: String,
pub title: String,
pub content: String,
#[serde(skip_serializing_if = "Vec::is_empty")]
pub headings: Vec,
#[serde(skip_serializing_if = "Option::is_none")]
pub scope: Option,
#[serde(skip_serializing_if = "Vec::is_empty")]
pub tags: Vec,
}
```

#### JavaScript Loader

```javascript
// search-loader.js
const SUPPORTED_VERSIONS = {
min: "1.0.0",
max: "1.99.99"
};

export async function loadSearchIndex(url) {
const response = await fetch(url);
const index = await response.json();

// Version check
const version = index.version || "0.0.0";

if (compareVersions(version, SUPPORTED_VERSIONS.min) < 0) {
console.warn(`Migrating old search index v${version}`);
return migrate(index);
}

if (compareVersions(version, SUPPORTED_VERSIONS.max) > 0) {
throw new Error(
`Search index v${version} is too new. ` +
`Max supported: ${SUPPORTED_VERSIONS.max}`
);
}

return index;
}

function compareVersions(a, b) {
const [aMajor, aMinor, aPatch] = a.split('.').map(Number);
const [bMajor, bMinor, bPatch] = b.split('.').map(Number);

if (aMajor !== bMajor) return aMajor - bMajor;
if (aMinor !== bMinor) return aMinor - bMinor;
return aPatch - bPatch;
}
```

### Configuration

```toml
[output.htmx.search]
enabled = true
index-format = "1.0" # Major.minor
backend = "lunr" # lunr | meilisearch | algolia | custom
include-code = false # Index code blocks
include-headings = true # Index headings
min-chars = 2 # Minimum query length
stop-words = true # Filter common words
```

## Size Optimization

### Content Truncation

```rust
impl SearchPage {
pub fn from_chapter(chapter: &Chapter, config: &SearchConfig) -> Self {
let content = if config.truncate {
truncate_content(&chapter.content, config.max_length)
} else {
chapter.content.clone()
};

Self {
path: chapter.path.clone(),
title: chapter.name.clone(),
content,
..Default::default()
}
}
}

fn truncate_content(content: &str, max_length: usize) -> String {
if content.len() <= max_length {
return content.to_string();
}

// Truncate at word boundary
let truncated = &content[..max_length];
if let Some(last_space) = truncated.rfind(' ') {
truncated[..last_space].to_string()
} else {
truncated.to_string()
}
}
```

### Compression

```javascript
// For large indexes, use compression
async function loadCompressedIndex(url) {
const response = await fetch(url);
const blob = await response.blob();

// Decompress if gzipped
if (url.endsWith('.gz')) {
const ds = new DecompressionStream('gzip');
const decompressed = blob.stream().pipeThrough(ds);
const text = await new Response(decompressed).text();
return JSON.parse(text);
}

return response.json();
}
```

## Consequences

### Positive

- Clear upgrade path with versioning
- Backward compatible minor changes
- Self-documenting format
- Multiple backend support

### Negative

- Version checking adds complexity
- Migration code must be maintained
- Larger index with more fields

### Mitigation

- Clear version constants
- Migration functions are isolated
- Optional fields reduce size

## Alternatives Considered

### No Versioning

Just change the format as needed.

**Rejected** because:
- No way to detect incompatibility
- Breaks existing deployments
- No migration path

### Breaking Changes Only

Only use major versions, always break.

**Rejected** because:
- Forces unnecessary upgrades
- Loses backward compatibility
- Bad user experience

### Separate Index Files

Different file per backend.

**Rejected** because:
- Complicates build process
- Multiple files to manage
- Harder to switch backends

## References

- [Semantic Versioning](https://semver.org/)
- [JSON Schema](https://json-schema.org/)
- [Lunr.js Serialization](https://lunrjs.com/guides/serialisation.html)
- [Meilisearch Documents](https://docs.meilisearch.com/learn/core_concepts/documents.html)
- [Algolia Record Format](https://www.algolia.com/doc/guides/sending-and-managing-data/prepare-your-data/)