InternalParquetRecordWriter - Too much memory used
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
Our logs are full of entries starting with `InternalParquetRecordWriter - Too much memory used: (...)`. We shall investigate why this is logged and whether it is a logging issue or some regression in our memory consumption.
**Reporter**: [Gabor Szadovszky](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=gszadovszky) / @gszadovszky
**Note**: *This issue was originally created as [PARQUET-1833](https://issues.apache.org/jira/browse/PARQUET-1833). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating InternalParquetRecordWriter and the code that emits the “Too much memory used” log entry. Reproduce or trace the condition that triggers it, then determine whether the message is caused by incorrect logging or a regression in memory consumption. Done means the cause is identified and the issue has a clear corrective direction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100