implicitly defining DEBUG mode in MessageColumnIO causes 80% performance overhead
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
**parquet-column** jar leverages slf4j and log4j as default logger, neglecting to define a log4j configuration, defaults to **DEBUG** log level.
```java
public class MessageColumnIO extends GroupColumnIO {
private static final Logger LOG = LoggerFactory.getLogger(MessageColumnIO.class);
private static final boolean DEBUG = LOG.isDebugEnabled(); // <------
}
```
this "magic behavior" defaults parquet library to be in DEBUG mode, without any notification or warnings. Unfortunately, the **RecordConsumerLoggingWrapper** implementation generates 5x performance overhead in comparison to the **MessageColumnIORecordConsumer** implementation, causing a massive hit in performance and wasteful server utilization.
IMHO there are two things that could prevent such issue:
- printing a message to STDOUT notifying about DEBUG mode being set to active.
- defaulting to **MessageColumnIORecordConsumer** implementation, and waiting for explicit configuration to define DEBUG mode, and use **RecordConsumerLoggingWrapper**.
In the past 2 years, this issue probably cost my company 50,000$ in excessive cloud costs!
**Reporter**: [Elad Yosifon](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=elad_yosifon)
**Note**: *This issue was originally created as [PARQUET-2036](https://issues.apache.org/jira/browse/PARQUET-2036). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 MessageColumnIO 开始,检查 LOG.isDebugEnabled() 如何在 MessageColumnIORecordConsumer 和 RecordConsumerLoggingWrapper 之间进行选择。跟踪 logger 配置以及现有的性能测试或基准测试;完成的标准是 DEBUG 未被隐式启用,并且普通 consumer 在未显式配置的情况下不会产生所报告的开销。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering, performance
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 30/100