dolthub / dolthub/doltlite

sqlite_dbpage: synthesized page 1 only; deeper pages reject as unsupported

Open
#947 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
268
Forks
18
Avg merge
2h 26m
Merged PRs (30d)
454

Description

## Summary
DoltLite exposes \`sqlite_dbpage\` for tools that expect it, but the implementation diverges from stock SQLite. Page 1 returns a synthesized header populated with derived values; any other \`pgno\` now errors instead of returning EOF (the prior behavior silently returned nothing).

## Why it diverges
Stock SQLite's database file is a linear array of fixed-size pages (default 4 KiB). \`sqlite_dbpage\` is a thin SQL accessor over that array — \`pgno=N\` returns bytes \`(N-1)*pageSize .. N*pageSize\`.

DoltLite has no pages. Storage is a tree of content-addressed chunks; chunks are variable-sized and addressed by hash, not by sequential page number. There is no meaningful \`pgno=47\`.

## What's implemented (src/doltlite_dbpage.c)
- \`pgno=1\` (or full scan): synthesizes a 4 KiB page-1 header with realistic-looking fields:
- \`SQLite format 3\` magic
- \`pageSize = 4096\`
- \`changeCounter\` derived from the session head commit hash
- \`pageCount = user-table-count\` (deliberately not the chunk count)
- \`schemaCookie\` derived from the head catalog hash
- \`largestRootPgno\` = max(table iTable)
- \`pgno > 1\`: \`SQLITE_ERROR\` with message \"doltlite: sqlite_dbpage only supports pgno=1 (content-addressed chunk store has no page layout)\".

## Impact on tools
- \`sqlite3_analyzer\`, page-utilization scripts, raw page-level backup tools: do not work. They expect to walk the full page range and will get an error at the first \`pgno > 1\`.
- Header-only inspection (e.g. detect SQLite format version, page size, schema cookie): works against page 1.
- Anything that scans \`sqlite_dbpage\` for diagnostics: gets a single row.

## Why the error now (was EOF before)
Silently returning zero rows for \`pgno > 1\` made tools think every page past page 1 was missing or empty, which produced misleading "database has 1 page" reports without any signal that doltlite couldn't answer. The error makes the unsupportedness explicit.

## Workarounds
- For database-size monitoring: use \`SELECT count(*)\` on user tables or query the chunk store via doltlite-specific functions instead.
- For page-level forensics: this isn't a workflow doltlite supports.

## Future options
1. Stay where we are. \`sqlite_dbpage\` covers \"is this a SQLite-format file? what's the schema cookie?\" and explicitly rejects the rest.
2. Build a chunk-store-flavored vtable (e.g. \`doltlite_chunks(hash, size, data)\`) for the diagnostic use cases. Doesn't pretend to be \`sqlite_dbpage\`.
3. Synthesize a believable page-mapping over the chunk store so \`sqlite3_analyzer\` reports something. Most work, dubious payoff — the numbers would be lies aligned to look real.

Filing here so the divergence is visible to anyone who finds doltlite via the standard SQLite tooling.

## Related
- Deep-review finding **P8** (the substance of the divergence).
- Deep-review finding **P9** (the registration plumbing): fixed in the same PR that filed this issue.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/doltlite_dbpage.c and review the referenced P8 and P9 findings, including the related registration changes. The issue documents current behavior and lists three possible future directions, but does not select an implementation target; completion criteria therefore need to be agreed before coding can begin.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, sqlite
Domain
databases
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.