apache / apache/texera

Read a Parquet file as a source

Open
#8,511 1 comment 0 reactions 1 assignee Claimed by @kz930 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

Texera reads CSV, JSONL, Arrow and plain text off disk, but not Parquet, which is the format most tables in a data-science workflow are already stored in. A user holding one has to convert it first, and a conversion to CSV loses the types the file was written with: the column that was an INTEGER comes back as text for the schema to guess at again.

A scan source that reads Parquet closes that. It is the same shape as the CSV and Arrow sources, a `ScanSourceOpDesc` declaring `fileTypeName = "Parquet"`, and it reads its schema off the file's own footer rather than inferring one, so the types survive. `pyarrow` is already a declared dependency, and Arrow's source shows how a binary format is read.

The parity harness covers it the moment the format is registered: `SourceCategoryRunner` maps a declared `fileTypeName` to an encoder that writes a fixture in it, and Parquet needs one entry added to `encoderByFileType`. This is the first operator to exercise that branch, the four already there being covered by an encoder that existed.

Not part of #8325: nothing here makes a workflow exportable. It does ship standalone code, so it comes after the trait #8327 introduces, and its fixture needs the harness that issue builds.

### Task Type

- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [X] Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.