apache / apache/parquet-java

implicitly defining DEBUG mode in MessageColumnIO causes 80% performance overhead

Open
#1,622 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Critical Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

**parquet-column** jar leverages slf4j and log4j as default logger, neglecting to define a log4j configuration, defaults to **DEBUG** log level.

 
```java

public class MessageColumnIO extends GroupColumnIO {
private static final Logger LOG = LoggerFactory.getLogger(MessageColumnIO.class);

private static final boolean DEBUG = LOG.isDebugEnabled(); // <------
}
```
 

this "magic behavior" defaults parquet library to be in DEBUG mode, without any notification or warnings. Unfortunately, the **RecordConsumerLoggingWrapper** implementation generates 5x performance overhead in comparison to the **MessageColumnIORecordConsumer** implementation, causing a massive hit in performance and wasteful server utilization.

 

IMHO there are two things that could prevent such issue:
- printing a message to STDOUT notifying about DEBUG mode being set to active.
- defaulting to **MessageColumnIORecordConsumer** implementation, and waiting for explicit configuration to define DEBUG mode, and use **RecordConsumerLoggingWrapper**.

 

In the past 2 years, this issue probably cost my company 50,000$ in excessive cloud costs!

**Reporter**: [Elad Yosifon](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=elad_yosifon)

**Note**: *This issue was originally created as [PARQUET-2036](https://issues.apache.org/jira/browse/PARQUET-2036). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with MessageColumnIO and inspect how LOG.isDebugEnabled() selects between MessageColumnIORecordConsumer and RecordConsumerLoggingWrapper. Trace the logger configuration and any existing performance tests or benchmarks; done means DEBUG is not implicitly enabled and the normal consumer avoids the reported overhead unless explicitly configured.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.