FasterXML / FasterXML/jackson-dataformats-binary

Add Apache Parquet format backend (`jackson-dataformat-parquet`)

Open
#766 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
347
Forks
156
Avg merge
3d 3h
Merged PRs (30d)
22

Description

Apache Parquet has become the de-facto interchange format for analytical/columnar data (Spark, Trino, Athena, DuckDB, Arrow, Iceberg/Delta table formats all read and write it).

Today, JVM applications that want to produce or consume Parquet go through the parquet-java libraries directly (`org.apache.parquet:parquet-hadoop` and friends), which means hand-writing `WriteSupport`/`ReadSupport` plumbing or converting through another object model (`parquet-avro`, `parquet-protobuf`). Point solutions exist for narrow slices (see prior art below), but there is no first-class Jackson backend: no `JsonFactory`-level parser/generator pair giving bidirectional streaming, databind (POJOs, `Map`s) and tree-model (`JsonNode`) access under the `ObjectMapper` programming model applications already use for JSON, CBOR, Avro, etc.

Additionally, we've internally observed that adding Hadoop as a dependency ends up blowing the dependencies' size, forcing a lot of teams to drop AWS Lambdas (or using containerised lambdas)

A `jackson-dataformat-parquet` backend would close that gap: the same `ObjectMapper`
programming model, applied to the dominant analytics file format.

## Prior art

* https://github.com/getyourguide/parquet-json — Parquet `WriteSupport` driven by Jackson `JsonNode`:
overlaps the write path proposed here (no read path, and no `JsonFactory`-level
parser/generator or databind integration)
* https://github.com/jerolba/parquet-carpet — Java-records read/write binding over parquet-java,
demonstrating demand for a friendlier binding layer (records only; not a general
Jackson backend)
* `parquet-avro` in parquet-java itself — the "bind through another object model"
pattern this module would generalize to Jackson's
* Searched this tracker, `jackson-future-ideas` and the org: no existing
Parquet request (only a passing mention in [#9](https://github.com/FasterXML/jackson-future-ideas/issues/9)).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.