apache / apache/datafusion-comet
Consistent debug logging levels and messages
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
To date Comet has been inconsistent with its debug logging. There are a handful of `logWarning`s for exceptional cases and minimal `logInfo` and `logDebug`. I know personally I try to use printing to help with development, and often strip it all back out before submitting a PR. Maybe a subset of that should be kept.
As users deploy Comet in production, we should make it easier to debug Comet behavior in Spark clusters.
### Describe the potential solution
This is mostly for discussion right now, but I think we should try to make Comet's logging consistent with Spark and Iceberg. We don't want it to be too verbose, but some info about plan conversation, plan distribution/serialization, and execution would be helpful.
I'm not sure if we should try to break this into smaller tasks like checking individual classes or modules to make it more manageable, or if the first task is to try to codify guidance into the developer guide.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.