apache / apache/datafusion-comet
Discuss `spark.comet.enabled` parameter behavior
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
The Rust log line
```
INFO core/src/lib.rs: Comet native library version initialized
```
is emitted even when Comet is disabled at the session level (e.g. `spark.comet.enabled=false`) or when the plugin bails out on the off-heap gate. The message is misleading because it implies Comet is running
when in fact no plan is being accelerated.
```
scala> spark.conf.get("spark.comet.enabled")
res0: String = false
scala> spark.read.parquet(s3://dataset/date=2026-07-21/hour=10/").limit(1).explain()
== Physical Plan ==
CollectLimit 1
+- *(1) ColumnarToRow
+- FileScan parquet
scala> :quit
26/08/05 15:01:17 INFO core/src/lib.rs: Comet native library version 1.0.0-apple initialized <<<<< should not be here
```
## Steps to reproduce
1. Register the Comet plugin: `--conf spark.plugins=org.apache.spark.CometPlugin`.
2. Disable Comet at the session level: `--conf spark.comet.enabled=false` (or leave `spark.memory.offHeap.enabled=false` and `spark.comet.exec.onHeap.enabled=false` so the plugin's off-heap check fails).
3. Start a Spark shell / submit a job.
4. Observe the driver / executor logs still contain `INFO core/src/lib.rs: Comet native library version ... initialized`.
Contributor guide
Research direction
Start by tracing the initialization path referenced by core/src/lib.rs and the handling of spark.comet.enabled, spark.memory.offHeap.enabled, and spark.comet.exec.onHeap.enabled. Reproduce the disabled-session and failed off-heap-gate cases, then verify that the native-library message no longer claims Comet is initialized when no plan is accelerated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100