Handle oversized page content during embedding sync

Summary

bigbrain sync --json can fail the entire sync when one changed page's embedding input exceeds the OpenAI embeddings token limit. In the June 11 nightly maintenance run for the brain workspace, OpenAI rejected input[22] with the 8192-token maximum, so sync exited before producing JSON and did not report page/link/embedding counts.

Reproduction

From /Users/harryberg/projects/brain:

BIGBRAIN_STATE_ROOT=/Users/harryberg/projects/brain/.bigbrain-state/brains bigbrain sync --json

Observed on 2026-06-11T03:31:18Z:

(node:74399) ExperimentalWarning: SQLite is an experimental feature and might change at any time
OpenAI embeddings failed: 400 {
  "error": {
    "message": "Invalid 'input[22]': maximum input length is 8192 tokens.",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}

Previous run on 2026-06-10T06:39:43Z succeeded with 361 indexed pages, 1923 indexed links, and 0 embeddings generated. The June 11 health command still succeeded separately and reported 379 pages, backlink coverage 372, and 81 findings, so the workspace is readable; the blocker is embedding generation for oversized input.

Likely Cause

src/bigbrain/sync.js currently embeds one text per changed page:

const texts = pagesNeedingEmbeddings.map((page) => `${page.title}\n\n${page.compiledTruth}`);
const vectors = await embedder(texts, config.openaiEmbeddingModel, apiKey);

A single large compiledTruth can exceed the embedding model input limit. Because the whole batch is sent in one request and the error is thrown from src/bigbrain/openai.js, one oversized page aborts the whole sync.

This aligns with the existing TODO items to add a chunk-level index and extend the embeddings table to support multiple chunks per page.

Expected Behavior

Sync should not fail globally because one page is too large for one embedding input. Acceptable fixes include:

• Chunk changed pages before embedding and store multiple chunk rows per page.
• Temporarily truncate or summarize overly long embedding inputs with a clear warning.
• Isolate embedding failures per page so indexing and link sync still complete and JSON output reports skipped/failed embeddings.
• Include the offending page slug/chunk in the error or warning so remediation is actionable.

Acceptance Criteria

• bigbrain sync --json completes indexing and link refresh even when one page exceeds the embedding input limit.
• JSON output includes enough detail to distinguish indexed pages/links from generated/skipped/failed embeddings.
• Oversized page handling is covered by a regression test.
• The user-facing error/warning names the page slug or chunk that caused the problem.

Start with src/bigbrain/sync.js and src/bigbrain/openai.js, then review the existing TODOs for a chunk-level index and multiple embeddings per page. Reproduce with BIGBRAIN_STATE_ROOT=... bigbrain sync --json using an oversized page. Done means sync completes indexing and link refresh, JSON distinguishes embedding outcomes, and a regression test and warning identify the page slug or chunk.