Parquet, categoricals and data-types
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 755
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
I have noticed that there are some differences between the parquet and the arff files (e.g. the classes integer and double can be different between the two formats), furthermore the arrow-reader uses non-standard metadata to encode the categoricals (see this issue: https://github.com/duckdb/duckdb/issues/3309#issuecomment-1087755900).
The arrow library however is really unusable in R (multiple people reported that), I am not sure how it would be in julia or Java (?)
Also the "features" metadata currently does not provide enough information to ensure that the parsed arff files and the parsed parquet files are really identical (by converting the columns)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the Parquet and ARFF representations, then inspect the Arrow reader's categorical metadata and the current "features" metadata. Define the information needed for parsed ARFF and Parquet files to be identical after column conversion, and verify that the comparison covers integer, double, and categorical types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, julia, r
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100