Read a Parquet file as a source
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Task Summary
Texera reads CSV, JSONL, Arrow and plain text off disk, but not Parquet, which is the format most tables in a data-science workflow are already stored in. A user holding one has to convert it first, and a conversion to CSV loses the types the file was written with: the column that was an INTEGER comes back as text for the schema to guess at again.
A scan source that reads Parquet closes that. It is the same shape as the CSV and Arrow sources, a `ScanSourceOpDesc` declaring `fileTypeName = "Parquet"`, and it reads its schema off the file's own footer rather than inferring one, so the types survive. `pyarrow` is already a declared dependency, and Arrow's source shows how a binary format is read.
The parity harness covers it the moment the format is registered: `SourceCategoryRunner` maps a declared `fileTypeName` to an encoder that writes a fixture in it, and Parquet needs one entry added to `encoderByFileType`. This is the first operator to exercise that branch, the four already there being covered by an encoder that existed.
Not part of #8325: nothing here makes a workflow exportable. It does ship standalone code, so it comes after the trait #8327 introduces, and its fixture needs the harness that issue builds.
### Task Type
- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [X] Other
Contributor guide
Assessment
This issue has not been assessed yet.