[Proposal] feat(table): compute and write table statistics (NDV) as Puffin StatisticsFile
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 121
Description
### Feature Request / Improvement
## Proposal: Table statistics computation (NDV via Theta sketches)
### Motivation
iceberg-go can *read* and *track* `StatisticsFile` metadata, but it cannot **compute** or **write** it. Tables created or maintained by iceberg-go therefore carry no NDV (number-of-distinct-values) statistics, so query engines that use them for cost-based planning (join ordering, cardinality estimation) get nothing from Go-managed tables. This proposes the missing **producer**, closing a parity gap with the Java implementation.
### Current state (what already exists)
Modeling and lifecycle are in place — only the producer is missing:
- `StatisticsFile` / `BlobMetadata` types and the `apache-datasketches-theta-v1` blob-type constant — `table/statistics.go`, `puffin/puffin.go`
- Metadata registration — `SetStatisticsUpdate` / `RemoveStatisticsUpdate` (`table/updates.go`), builder support in `table/metadata.go`
- Lifecycle — statistics pruned on `RemoveSnapshots`, deleted on `ExpireSnapshots`, included in orphan-file cleanup
What is missing: nothing computes a Theta sketch, and the only Puffin *write* path that emits a blob today is the deletion-vector writer (`table/dv/dv_writer.go`). There is no "analyze this snapshot → produce a `StatisticsFile`" entry point.
### Goals
- Compute per-column NDV for a snapshot as Theta sketches.
- Serialize sketches in a form **byte-compatible with Apache DataSketches**, so blobs written by iceberg-go are readable by Java/Spark and vice versa.
- Write a valid Puffin `StatisticsFile` (correct `snapshot-id` / `sequence-number` footer) and register it via the existing `SetStatistics` update.
- Expose a CLI entry point to compute statistics for a table/snapshot.
### Design sketch
1. **Theta sketch core** — pure-Go sketch (`Update` / `Union` / `Estimate`), no Iceberg dependencies.
2. **DataSketches-compatible serialization** — compact serialized form with Java-produced fixtures as the parity oracle (the correctness-critical piece).
3. **NDV compute pass** — scan a snapshot's data files, feed values into per-field-ID sketches, emit theta blob bytes keyed by field ID.
4. **Puffin `StatisticsFile` writer + registration** — assemble blobs into a Puffin file and commit through `SetStatistics`.
5. **CLI + docs** — a `compute-stats`-style command and reference docs.
### Proposed PR breakdown (6 PRs )
1. `feat(table): add Theta sketch core (update/union/estimate)` — pure algorithm + error-bound tests.
2. `feat(table): DataSketches-compatible Theta sketch serialization` — compact encode/decode + Java-fixture byte cross-check.
3. `feat(table): compute per-column NDV over a snapshot` — scan → per-field sketches → blob bytes.
4. `feat(table): write StatisticsFile and register via SetStatistics` — Puffin write + commit (end-to-end).
5. `feat(cli): add compute-stats command` — CLI wired to (4), text/JSON output.
6. `docs(website): document table statistics` + read-side surface — docs and a compute→reload→verify integration test.
### Open questions
Theta sketch serialization: The blob must be byte-compatible with Apache DataSketches so Java/Spark can read it. I don't see a maintained Go port of the compact Theta format, so my lean is to implement the compact serialization in-tree (bounded, and spec'd by DataSketches) and validate byte-for-byte against Java-produced fixtures — the same approach the DV code uses against Java. Flagging in case there's a preferred dependency I should use instead.
Contributor guide
Assessment
This issue has not been assessed yet.