apache / apache/parquet-java

InternalParquetRecordWriter - Too much memory used

Open
#2,478 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Major Type: test
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Our logs are full of entries starting with `InternalParquetRecordWriter - Too much memory used: (...)`. We shall investigate why this is logged and whether it is a logging issue or some regression in our memory consumption.

**Reporter**: [Gabor Szadovszky](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=gszadovszky) / @gszadovszky

**Note**: *This issue was originally created as [PARQUET-1833](https://issues.apache.org/jira/browse/PARQUET-1833). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating InternalParquetRecordWriter and the code that emits the “Too much memory used” log entry. Reproduce or trace the condition that triggers it, then determine whether the message is caused by incorrect logging or a regression in memory consumption. Done means the cause is identified and the issue has a clear corrective direction.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.