apache / apache/parquet-java

Convert JSON to Parquet directly

Open
#2,503 1 comment 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Minor Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

In order to convert JSON payload to Parquet, the "established" way seems to be to go first through a JSON to Avro conversion, provided the JSON payload is covered by a JSON-Schema or part of a OpenAPI definition. The rest of the process relies on the parquet-avro implementation.

Since JSON payload can be loaded to a generic JsonNode (jackson library) and since a JSON payload can be described and checked against a JSON-Schema or an OpenAPI definition, wouldn't it be theoretically possible to convert a JSON payload to a Parquet "record" directly (somehow implementing a similar approach than parquet-avro but this a JsonNode as a data holder)?

**Reporter**: [Emmanuel Brard](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=ebrard) / @ebrard

**Note**: *This issue was originally created as [PARQUET-1880](https://issues.apache.org/jira/browse/PARQUET-1880). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the existing parquet-avro implementation and how Jackson JsonNode represents and validates JSON against JSON-Schema or OpenAPI definitions. Define the scope and design for producing a Parquet record directly from JSON, including what constitutes a complete conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.