apache / apache/iceberg

Add data-file byte totals to manifest_list entries, so table size can be read without scanning manifest bodies

Open
#17,743 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Feature Request / Improvement

`manifest_list` entries summarize counts and rows, but not bytes:

```
added_files_count existing_files_count deleted_files_count
added_rows_count existing_rows_count deleted_rows_count
manifest_length <- the manifest file's own size, not the data it describes
```

Every `data_file` already carries `file_size_in_bytes`, so the number exists — it just isn't rolled up where it can be read cheaply. Getting a table's reachable bytes today means opening **every** manifest body and summing per entry, which is decompression-bound and almost entirely redundant: manifests are immutable and inherited largely unchanged between snapshots, so the same bytes are re-inflated every run.

## Proposal

Three optional fields on `manifest_file`, mirroring the existing triplets:

```
added_files_size_in_bytes
existing_files_size_in_bytes
deleted_files_size_in_bytes
```

Zero write cost — the writer already holds every `file_size_in_bytes`. Backward compatible: absent means readers fall back to reading bodies.

## Why snapshot summary `total-files-size` doesn't cover it

It gives one snapshot's total, and summing across live snapshots double counts files shared between them. Manifests are the natural dedup unit, since a manifest is inherited wholesale — dedup by `manifest_path`, then sum. That composes; per-snapshot totals don't.

Useful well beyond one use case: cost attribution, compaction and retention planning, catalog-wide dashboards. #14803 and #14820 added delete-file size totals to `PartitionsTable`; this is the same idea in the manifest list, readable without a scan.

I found no prior discussion (searched `added_files_size`, `manifest_list` + bytes, spec-labelled size proposals) — happy to be pointed at it. As a spec change this presumably wants a dev@ thread and spec PR; filing here to gauge support.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the manifest_list and manifest_file entry definitions alongside data_file.file_size_in_bytes. Confirm the proposed added, existing, and deleted size fields preserve backward compatibility and determine the required spec changes. Discuss the design on dev@ and use the resulting spec PR as the completion reference.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.