apache / apache/parquet-java

Could we provide a potentially larger InternalParquetRecordWriter.getDataSize

Open
#3,032 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

The following code currently has getDataSize as an estimated value. The Iceberg rolling file write operation relies on this method, which may result in writing files that are much smaller than expected.
```JAVA
/**
* @return the total size of data written to the file and buffered in memory
*/
public long getDataSize() {
return lastRowGroupEndPos + columnStore.getBufferedSize();
}
```

Could we provide a potentially larger getDataSize? I can't think of any downsides at the moment.

### Component(s)

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.