apache / apache/iceberg

File Type Support

Open
#17,919 0 comments 1 reaction 0 assignees View on GitHub
proposal
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Proposed Change

Authors: Talat Uyarer, Alex Stephen, Sung Yun, Gaurav Soni

Add file type to Iceberg v4

Add a new file primitive type to the Iceberg v4 spec for referencing a range of bytes, stored either inline or in an external object.

The type and its value semantics are defined by the Parquet FILE logical type (apache/parquet-format#585); The Iceberg spec covers the schema representation, field-ID reservation, statistics behavior, and file-format mappings (Parquet/Avro/ORC).

Spec PR #17918
Reference Impl PR #17821

### Proposal document

https://docs.google.com/document/d/1BdA4G-38GoVcpX6UY4UI-QA1MvrDnH6uSKeCJBadILQ/edit?tab=t.pnp3scifr79p#heading=h.v4a92wm7mz0o

### Specifications

- [x] Table
- [ ] View
- [ ] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other

Contributor guide

Open the contributing guide

Research direction

Start with the proposal document, then compare Iceberg v4 spec PR #17918 and reference implementation PR #17821. Review the checked Table item and the unchecked View, REST, Puffin, and Encryption items; done means the file primitive is consistently covered in the applicable specifications and implementation, including schema, field-ID, statistics, and format mappings.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.