apache / apache/iceberg

[REST Scan Planning] Missing sequence number on `DataFile` object

Open Beginner friendly
#17,319 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Apache Iceberg version

None

### Query engine

None

### Please describe the bug 🐞

The `DataFile` object, returned by the scan-planning API is missing `sequence-number`, making it impossible to honor `_last_updated_sequence_number` queries with the scan-planning API.

```yaml
DataFile:
allOf:
- $ref: '#/components/schemas/ContentFile'
type: object
required:
- content
properties:
content:
type: string
const: "data"
first-row-id:
type: integer
format: int64
description: "The first row ID assigned to the first row in the data file"
column-sizes:
allOf:
- $ref: '#/components/schemas/CountMap'
description: "Map of column id to total count, including null and NaN"
value-counts:
allOf:
- $ref: '#/components/schemas/CountMap'
description: "Map of column id to null value count"
null-value-counts:
allOf:
- $ref: '#/components/schemas/CountMap'
description: "Map of column id to null value count"
nan-value-counts:
allOf:
- $ref: '#/components/schemas/CountMap'
description: "Map of column id to number of NaN values in the column"
lower-bounds:
allOf:
- $ref: '#/components/schemas/ValueMap'
description: "Map of column id to lower bound primitive type values"
upper-bounds:
allOf:
- $ref: '#/components/schemas/ValueMap'
description: "Map of column id to upper bound primitive type values"
```

Would need an optional:
```yaml
sequence-number:
type: integer
format: int64
description: "The sequence number assigned to the manifest entry or manifest file that this data file belongs to"
```

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by locating the REST scan-planning API schema definition for the DataFile object shown in the issue. Add the optional sequence-number property with the specified type and description, then inspect the scan-planning response path to confirm the value is exposed for DataFile entries. Done means clients can use the returned sequence number to honor _last_updated_sequence_number queries.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.