apache / apache/datafusion-comet

Consistent debug logging levels and messages

Open
#3,445 1 comment 4 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

To date Comet has been inconsistent with its debug logging. There are a handful of `logWarning`s for exceptional cases and minimal `logInfo` and `logDebug`. I know personally I try to use printing to help with development, and often strip it all back out before submitting a PR. Maybe a subset of that should be kept.

As users deploy Comet in production, we should make it easier to debug Comet behavior in Spark clusters.

### Describe the potential solution

This is mostly for discussion right now, but I think we should try to make Comet's logging consistent with Spark and Iceberg. We don't want it to be too verbose, but some info about plan conversation, plan distribution/serialization, and execution would be helpful.

I'm not sure if we should try to break this into smaller tasks like checking individual classes or modules to make it more manageable, or if the first task is to try to codify guidance into the developer guide.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.