apache / apache/iceberg-rust

manifests metadata table reports file counts in both data and delete columns

Open
#2,812 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

The `manifests` metadata table writes every manifest's added/existing/deleted counts into **both** the `*_data_files_count` and `*_delete_files_count` columns, regardless of the manifest's content type — a data manifest reports phantom delete-file counts, and a delete manifest reports phantom data-file counts.

The reference implementation gates each column by content type (`ManifestsTable.java`, `manifestFileToRow`):

```java
manifest.content() == ManifestContent.DATA ? manifest.addedFilesCount() : 0, // added_data_files_count
manifest.content() == ManifestContent.DELETES ? manifest.addedFilesCount() : 0, // added_delete_files_count
```

In `crates/iceberg/src/inspect/manifests.rs` the six `append_value` sites use the same `ManifestFile` counts unconditionally. Easy to see with a snapshot containing one data and one deletes manifest: both rows show identical counts in all six columns.

Found while exploring #823. I have a fix with tests ready; PR incoming.

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/inspect/manifests.rs at the six append_value sites and compare their behavior with the content-type rules shown from ManifestsTable.java. Reproduce with a snapshot containing one data and one deletes manifest, then run the relevant manifest inspection tests; done means each row reports counts only in its matching data or delete columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
39/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.