InternalParquetRecordWriter flushRowGroupToStore not actually writing to file
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Description:
The flushRowGroupToStore method in InternalParquetRecordWriter purports to "Flush mem columnStore to file," but no data is actually written to the file system until the close method is invoked. This behavior is unexpected, as one would anticipate data being flushed.
Expected Result:
Data should be written to the file when flushRowGroupToStore is invoked, effectively flushing the in-memory column store contents to the file.
Actual Result:
File contents remain empty until the close method is called, where parquetFileWriter.end is invoked to write the data.
Version : parquet-hadoop 1.13.1
### Component(s)
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading InternalParquetRecordWriter, especially flushRowGroupToStore and close, then trace how parquetFileWriter.end writes buffered data. Confirm the expected flush boundary and add or update coverage for file contents after flushRowGroupToStore; done means the file contains the flushed row group before close is invoked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100