getdozer / getdozer/dozer

Support additional file formats for connections. [`.arrow`,`.json`, `.ndjson`]

Open
#1,437 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.6k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Currently you can only upload files that are in either parquet, or csv. It is not always be practical to convert your datasets to this format before uploading.

**Describe the solution you'd like**
Dozer should support a few additional common data formats. Targeting the ones supported by many other data processing tools would make sense.

Some of the common data formats that I'd like supported.
- json
- ndjson (newline delimited json)
- arrow (Apache Arrow IPC format. AKA `.feather, .ipc`)

**Describe alternatives you've considered**
Only alternative is to use an external tool to convert datasets into csv or parquet.

**Additional context**
It looks like dozer is using `datafusion` for some of the connectors, so I think it'd be rather straightforward to support the same file formats that `datafusion` supports.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.