Support additional file formats for connections. [`.arrow`,`.json`, `.ndjson`]
- Dominant language
- Rust
- Stars
- 1.6k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Currently you can only upload files that are in either parquet, or csv. It is not always be practical to convert your datasets to this format before uploading.
**Describe the solution you'd like**
Dozer should support a few additional common data formats. Targeting the ones supported by many other data processing tools would make sense.
Some of the common data formats that I'd like supported.
- json
- ndjson (newline delimited json)
- arrow (Apache Arrow IPC format. AKA `.feather, .ipc`)
**Describe alternatives you've considered**
Only alternative is to use an external tool to convert datasets into csv or parquet.
**Additional context**
It looks like dozer is using `datafusion` for some of the connectors, so I think it'd be rather straightforward to support the same file formats that `datafusion` supports.
Contributor guide
Assessment
This issue has not been assessed yet.