apache / apache/parquet-java

toParquetMetadata method in ParquetMetadataConverter does not set dictionary page offset bit

Open
#2,901 0 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

toParquetMetadata method converts org.apache.parquet.hadoop.metadata.ParquetMetadata to org.apache.parquet.format.FileMetaData but this does not set the dictionary page offset bit in FileMetaData.

When a FileMetaData object is serialized while writing to the footer and then deserialized, the dictionary offset is lost as the dictionary page offset bit was never set.

PARQUET-1850  tried to fix this but it did only a partial fix.

It sets setDictionary_page_offset only if getEncodingStats are present
```java

if (columnMetaData.getEncodingStats() != null
&& columnMetaData.getEncodingStats().hasDictionaryPages())
{ metaData.setDictionary_page_offset(columnMetaData.getDictionaryPageOffset()); }
```
However, it should setDictionary_page_offset even when getEncodingStats are not present but encodings are present.

It should use the implementation in ColumnChunkMetatdata below:
```java

public boolean hasDictionaryPage() {
EncodingStats stats = getEncodingStats();
if (stats != null) {
return stats.hasDictionaryPages() && stats.hasDictionaryEncodedPages();
}

Set encodings = getEncodings();
return (encodings.contains(PLAIN_DICTIONARY) || encodings.contains(RLE_DICTIONARY));
}
```
So new change in ParquetMetadataCOnvertor should be like:

 
```java

if (columnMetaData.hasDictionaryPage()) { metaData.setDictionary_page_offset(columnMetaData.getDictionaryPageOffset()); }
```

**Reporter**: [Abhishek Dixit](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=abhishekd0907)
#### PRs and other links:
- [GitHub Pull Request #1340](https://github.com/apache/parquet-mr/pull/1340)

**Note**: *This issue was originally created as [PARQUET-2464](https://issues.apache.org/jira/browse/PARQUET-2464). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.