apache / apache/parquet-java

Allow for custom compression codecs

Offen
#1,988 6 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Component: Parquet Priority: Minor Type: enhancement
Vorherrschende Sprache
Java
Sterne
3.1k
Forks
1.6k
Ø Merge
3 T. 12 Std.
Gemergte PRs (30 T.)
33

Beschreibung

I understand that the list of accepted compression codecs is explicity limited to uncompressed, snappy, gzip, and lzo. (See parquet.hadoop.metadata.CompressionCodecName.java) Is there a reason for this? Or is there an easy workaround? On the surface it seems like an unnecessary restriction.

I ask because I have written a custom codec to implement encryption and I'm unable to use it with Parquet, which is a real shame because it is the main storage format I was hoping to use.

Other thoughts on how to implement encryption in Parquet with this limitation?

**Reporter**: [Steven Anton](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=santon)

**Note**: *This issue was originally created as [PARQUET-678](https://issues.apache.org/jira/browse/PARQUET-678). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start with parquet.hadoop.metadata.CompressionCodecName.java, the file named in the issue, and review the migration documentation linked in the issue note. Clarify whether the goal is custom codec support, an encryption workaround, or both; done should be a decided and documented approach that permits the requested use case.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
data-engineering
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.