[FEA] Support the Parquet FILE logical type for external/blob payloads
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## Is your feature request related to a problem? Please describe.
[apache/parquet-format#585](https://github.com/apache/parquet-format/pull/585) added a **`FILE` logical type** to the Parquet spec — merged **2026-07-26** as `19: FileType FILE` in `parquet.thrift`, with the field semantics in [`LogicalTypes.md`](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file).
`FILE` annotates a **group** representing a reference to a range of bytes, which may be stored inline in the value, elsewhere within the same Parquet file, or in an external file. Every field is optional, in both the schema and the data; a group need only define the fields it uses:
| field | type | meaning |
|---|---|---|
| `uri` | STRING | RFC 3986 URI-reference. If not set, the value is a **self-reference** to the current Parquet file. |
| `offset` | INT64 | Start of the byte range. Required for a self-reference. |
| `size` | INT64 | Byte length. Required whenever `offset` is set. |
| `content_type` | STRING | RFC 2046 media type; defaults to `application/octet-stream`. |
| `checksum` | STRING | `:`; recognized algorithms are `ETAG`, `MD5`, `CRC32`, `CRC32C`, `SHA-256`. |
| `inline` | BYTE_ARRAY | The bytes themselves, stored in the value. When set, any locator fields are provenance only. |
Additional engine- or table-format-specific metadata must be stored *adjacent to* the group, not inside it.
It is the standards-track way to keep large unstructured payloads — images, audio, video, PDFs — addressable from a tabular column without inlining them. cuDF has no support for it and, as far as I can find, no tracking issue.
**Ecosystem status.** The spec is merged; the two PoC implementations referenced by the PR — [apache/parquet-java#3608](https://github.com/apache/parquet-java/pull/3608) and [apache/arrow-rs#10109](https://github.com/apache/arrow-rs/pull/10109) — are both still open. pyarrow has nothing (checked 23.0.1 and 25.0.0).
## Why this matters for GPU data pipelines
Multimodal curation is the workload. A canonical image table looks like:
```
url string
image <- ~140 KB median
mime_type string
width,height int32
sha256 string
embedding list[1024]
```
Today we express the payload reference **by hand**: a JSON locator string per row carrying `{path, member, byte_offset, byte_size}`, parsed on the host, with byte-range gathers issued through fsspec. That works, but it is bespoke (no other engine can read our table without our parser), unvalidated (a stale path surfaces as a runtime error, not a null), and opaque to cuDF — payload resolution happens entirely off-GPU. `FILE` is that locator, standardized and typed.
Note that `FILE` also touches the binary gap: the `inline` field is a `BYTE_ARRAY`, and **cuDF has no binary column type at all** (#23492 — Arrow `binary`/`large_binary` unsupported, Parquet binary read as `string`). So inline `FILE` values need that work too, while the locator-only variants do not — which makes the reference-based path the cheaper starting point.
## Describe the solution you'd like
Staged, in value order:
1. **Read/round-trip fidelity.** Read a `FILE` column as a struct column and write it back without loss. Even with no payload resolution, this lets cuDF sit in a pipeline where another stage resolves payloads — and avoids the silent type-rewriting we already hit with binary columns.
2. **Payload resolution.** An opt-in read mode that follows `uri` + `offset` + `size` and materializes bytes, **batched and coalesced** rather than one request per row.
3. **Integration with hybrid scan / sparse-page I/O.** Resolving a `FILE` column for a *filtered subset* of rows is precisely the sparse-gather pattern rapidsai/cudf#23362 and rapidsai/cudf#23375 are building: plan byte ranges, coalesce them, issue them concurrently. `FILE` gives that machinery a typed, spec-blessed column to operate on.
4. **Writer support**
## Describe alternatives you've considered
- **Inline `large_binary`** — the obvious approach, but wont work for really large dtypes like videos that are coming down the pipeline
- **A plain string path column** — what we do now. No validation, no engine interop, and the reader cannot plan I/O.
- **Lance blob encoding** — works, and we use it. Our own measurements, offered as design input rather than as a claim about cuDF: blob encoding beats inline binary on **local NVMe** (~4× on `read_blobs`, 16.4K vs 4.0K img/s) but **loses over S3** (~500 img/s vs ~2,466 img/s for plain binary + row-id), because per-blob fetch RTT eats the API-level saving. The lesson we would carry into any cuDF `FILE` implementation: **the win is in request coalescing and planning, not in the encoding itself.** An implementation that issues one GET per row will be slower than inlining, regardless of how good the type is.
## Additional context
Priority is below sparse-page I/O (#23519) — that work unblocks us now; this is the schema story for the next 6–12 months. They converge at stage 3 above.
Related logical-type work for precedent: rapidsai/cudf#22312 (ingest of Parquet VARIANT columns), rapidsai/cudf#23251 (JSON reader output → VARIANT), rapidsai/cudf#23182 (VARIANT logical-type inspection). Related from us: #23492 (no binary type).
Thanks @mhaseeb123 for pointting to this parquet file format that is being worked on
Contributor guide
Research direction
Start by reading the Parquet FILE specification in parquet.thrift and LogicalTypes.md, then review the related cuDF issues #23492, #23362, #23375, #23519, #22312, #23251, and #23182. The issue does not name cuDF files or tests; completion would first require an agreed scope and then lossless FILE struct read/round-trip support, with later payload resolution and writer work explicitly staged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100