apache / apache/parquet-java

implicitly defining DEBUG mode in MessageColumnIO causes 80% performance overhead

未关闭
#1,622 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Parquet Priority: Critical Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

**parquet-column** jar leverages slf4j and log4j as default logger, neglecting to define a log4j configuration, defaults to **DEBUG** log level.

 
```java

public class MessageColumnIO extends GroupColumnIO {
private static final Logger LOG = LoggerFactory.getLogger(MessageColumnIO.class);

private static final boolean DEBUG = LOG.isDebugEnabled(); // <------
}
```
 

this "magic behavior" defaults parquet library to be in DEBUG mode, without any notification or warnings. Unfortunately, the **RecordConsumerLoggingWrapper** implementation generates 5x performance overhead in comparison to the **MessageColumnIORecordConsumer** implementation, causing a massive hit in performance and wasteful server utilization.

 

IMHO there are two things that could prevent such issue:
- printing a message to STDOUT notifying about DEBUG mode being set to active.
- defaulting to **MessageColumnIORecordConsumer** implementation, and waiting for explicit configuration to define DEBUG mode, and use **RecordConsumerLoggingWrapper**.

 

In the past 2 years, this issue probably cost my company 50,000$ in excessive cloud costs!

**Reporter**: [Elad Yosifon](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=elad_yosifon)

**Note**: *This issue was originally created as [PARQUET-2036](https://issues.apache.org/jira/browse/PARQUET-2036). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 MessageColumnIO 开始,检查 LOG.isDebugEnabled() 如何在 MessageColumnIORecordConsumer 和 RecordConsumerLoggingWrapper 之间进行选择。跟踪 logger 配置以及现有的性能测试或基准测试;完成的标准是 DEBUG 未被隐式启用,并且普通 consumer 在未显式配置的情况下不会产生所报告的开销。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering, performance
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
30/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。