apache / apache/parquet-java

Implement writing Parquet files into Cassandra natively

Open
#1,764 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Minor Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Writing Parquet files into Cassandra could allow parallel writes of multiple pages into different cells, and low latency reads with a persistent connection to C\*.

Each page could be written to separate C\* cells, with metadata written into a separate column family.

A possible way of implementing is:

- abstract ParquetFileWriter -> ParquetDataWriter. writeDictionaryPage, writeDataPage are abstract methods.
- ParquetFileWriter implements ParquetDataWriter, writing the data to Hadoop compatible files.
- ParquetCassandraWriter implements ParquetDataWriter, writing data to Cassandra
- for each page, metadata is written to Metadata CF, with key :::
- for each page, data is written to Data CF, with key :::
- footer is written to Metadata CF, with key

- abstract ParquetFileReader -> ParquetDataReader. readNextRowGroup, readFooter are abstract methods. Chunk will also need to be abstract.
- ParquetFileReader implements ParquetDataReader, reading from Hadoop compatible files.
- ParquetCassandraReader implements ParquetDataReader, reading from Cassandra

- ParquetDataWriter and ParquetDataReader are instantiated through reflection.

**Reporter**: [Issac Buenrostro](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=ibuenros)

**Note**: *This issue was originally created as [PARQUET-224](https://issues.apache.org/jira/browse/PARQUET-224). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.