apache / apache/parquet-java

Implement writing Parquet files into Cassandra natively

Open
#1,764 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Minor Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Writing Parquet files into Cassandra could allow parallel writes of multiple pages into different cells, and low latency reads with a persistent connection to C\*.

Each page could be written to separate C\* cells, with metadata written into a separate column family.

A possible way of implementing is:

- abstract ParquetFileWriter -> ParquetDataWriter. writeDictionaryPage, writeDataPage are abstract methods.
- ParquetFileWriter implements ParquetDataWriter, writing the data to Hadoop compatible files.
- ParquetCassandraWriter implements ParquetDataWriter, writing data to Cassandra
- for each page, metadata is written to Metadata CF, with key :::
- for each page, data is written to Data CF, with key :::
- footer is written to Metadata CF, with key

- abstract ParquetFileReader -> ParquetDataReader. readNextRowGroup, readFooter are abstract methods. Chunk will also need to be abstract.
- ParquetFileReader implements ParquetDataReader, reading from Hadoop compatible files.
- ParquetCassandraReader implements ParquetDataReader, reading from Cassandra

- ParquetDataWriter and ParquetDataReader are instantiated through reflection.

**Reporter**: [Issac Buenrostro](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=ibuenros)

**Note**: *This issue was originally created as [PARQUET-224](https://issues.apache.org/jira/browse/PARQUET-224). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the existing ParquetFileWriter and ParquetFileReader entry points and the proposed ParquetDataWriter and ParquetDataReader abstractions. Then investigate how Cassandra column families and persistent connections would fit the page, metadata, and footer layout described here. Done means native Cassandra reader and writer implementations, reflection-based instantiation, and verified parallel page writes and reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.