apache / apache/parquet-java

Implement writing Parquet files into Cassandra natively

Aperta
#1,764 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Component: Parquet Priority: Minor Type: enhancement
Lingua principale
Java
Stelle
3.1k
Fork
1.6k
Merge medio
3g 12h
PR unite (30g)
33

Descrizione

Writing Parquet files into Cassandra could allow parallel writes of multiple pages into different cells, and low latency reads with a persistent connection to C\*.

Each page could be written to separate C\* cells, with metadata written into a separate column family.

A possible way of implementing is:

- abstract ParquetFileWriter -> ParquetDataWriter. writeDictionaryPage, writeDataPage are abstract methods.
- ParquetFileWriter implements ParquetDataWriter, writing the data to Hadoop compatible files.
- ParquetCassandraWriter implements ParquetDataWriter, writing data to Cassandra
- for each page, metadata is written to Metadata CF, with key :::
- for each page, data is written to Data CF, with key :::
- footer is written to Metadata CF, with key

- abstract ParquetFileReader -> ParquetDataReader. readNextRowGroup, readFooter are abstract methods. Chunk will also need to be abstract.
- ParquetFileReader implements ParquetDataReader, reading from Hadoop compatible files.
- ParquetCassandraReader implements ParquetDataReader, reading from Cassandra

- ParquetDataWriter and ParquetDataReader are instantiated through reflection.

**Reporter**: [Issac Buenrostro](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=ibuenros)

**Note**: *This issue was originally created as [PARQUET-224](https://issues.apache.org/jira/browse/PARQUET-224). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start by reviewing the existing ParquetFileWriter and ParquetFileReader entry points and the proposed ParquetDataWriter and ParquetDataReader abstractions. Then investigate how Cassandra column families and persistent connections would fit the page, metadata, and footer layout described here. Done means native Cassandra reader and writer implementations, reflection-based instantiation, and verified parallel page writes and reads.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
cassandra, java
Ambito
databases
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.