implicitly defining DEBUG mode in MessageColumnIO causes 80% performance overhead
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
**parquet-column** jar leverages slf4j and log4j as default logger, neglecting to define a log4j configuration, defaults to **DEBUG** log level.
```java
public class MessageColumnIO extends GroupColumnIO {
private static final Logger LOG = LoggerFactory.getLogger(MessageColumnIO.class);
private static final boolean DEBUG = LOG.isDebugEnabled(); // <------
}
```
this "magic behavior" defaults parquet library to be in DEBUG mode, without any notification or warnings. Unfortunately, the **RecordConsumerLoggingWrapper** implementation generates 5x performance overhead in comparison to the **MessageColumnIORecordConsumer** implementation, causing a massive hit in performance and wasteful server utilization.
IMHO there are two things that could prevent such issue:
- printing a message to STDOUT notifying about DEBUG mode being set to active.
- defaulting to **MessageColumnIORecordConsumer** implementation, and waiting for explicit configuration to define DEBUG mode, and use **RecordConsumerLoggingWrapper**.
In the past 2 years, this issue probably cost my company 50,000$ in excessive cloud costs!
**Reporter**: [Elad Yosifon](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=elad_yosifon)
**Note**: *This issue was originally created as [PARQUET-2036](https://issues.apache.org/jira/browse/PARQUET-2036). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.