ParquetRecordWriter does not provide interface like getRowCount(),getRawDataSize() like org.apache.orc.Writer
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
Hive with orc can update the statistics like rowCount,rawDataSize after loading data to table. Hive with parquet cannot and need to use analyze command like "analyze table xxx compute statistics noscan" to update these two statistics info. The reason is ParquetRecordWriter used in hive does not provide interfaces like getRowCount(),getRawDataSize(). While org.apache.orc.Writer provides these [two interfaces](https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Writer.java#L68 ). Anyone knows how to get rowCount and rawDataSize in ParquetRecordWriter?
**Reporter**: [liyunzhang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=kellyzly)
**Note**: *This issue was originally created as [PARQUET-1117](https://issues.apache.org/jira/browse/PARQUET-1117). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating ParquetRecordWriter in this repository and compare its available interfaces with the org.apache.orc.Writer references linked in the issue. Check how Hive currently obtains rowCount and rawDataSize, then confirm that the requested statistics can be exposed without requiring the stated analyze table command.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100