apache / apache/parquet-java

Parquet pages for different columns cannot be read in parallel

Open
#2,358 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Major Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

All ColumnChunkPageReader instances use the same decompressor. 

```java

BytesInputDecompressor decompressor = options.getCodecFactory().getDecompressor(descriptor.metadata.getCodec());
return new ColumnChunkPageReader(decompressor, pagesInChunk, dictionaryPage);
```
The CodecFactory caches the decompressors for every codec type returning the same instance on every getCompressor(codecName) call. See the caching happening here:

```java

@Override
public BytesDecompressor getDecompressor(CompressionCodecName codecName) {
BytesDecompressor decomp = decompressors.get(codecName);
if (decomp == null){
decomp = createDecompressor(codecName); decompressors.put(codecName, decomp);
}
return decomp;
}
 
```
 

If multiple threads try to read the pages belonging to different columns, they run into thread

safety issues. This issue prevents increasing the throughput at which applications can read parquet data by parallelizing page reads. 

**Reporter**: [Samarth Jain](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=samarthjain) / @samarthjain
**Assignee**: [Samarth Jain](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=samarthjain) / @samarthjain
#### PRs and other links:
- [GitHub Pull Request #668](https://github.com/apache/parquet-mr/pull/668)
- [GitHub Pull Request #670](https://github.com/apache/parquet-mr/pull/670)

**Note**: *This issue was originally created as [PARQUET-1641](https://issues.apache.org/jira/browse/PARQUET-1641). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.