apache / apache/datafusion-comet

Discuss `spark.comet.enabled` parameter behavior

Open
#5,274 5 comments 0 reactions 0 assignees View on GitHub
bug priority:low
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

The Rust log line

```
INFO core/src/lib.rs: Comet native library version initialized
```

is emitted even when Comet is disabled at the session level (e.g. `spark.comet.enabled=false`) or when the plugin bails out on the off-heap gate. The message is misleading because it implies Comet is running
when in fact no plan is being accelerated.

```
scala> spark.conf.get("spark.comet.enabled")
res0: String = false

scala> spark.read.parquet(s3://dataset/date=2026-07-21/hour=10/").limit(1).explain()
== Physical Plan ==
CollectLimit 1
+- *(1) ColumnarToRow
+- FileScan parquet

scala> :quit
26/08/05 15:01:17 INFO core/src/lib.rs: Comet native library version 1.0.0-apple initialized <<<<< should not be here
```

## Steps to reproduce

1. Register the Comet plugin: `--conf spark.plugins=org.apache.spark.CometPlugin`.
2. Disable Comet at the session level: `--conf spark.comet.enabled=false` (or leave `spark.memory.offHeap.enabled=false` and `spark.comet.exec.onHeap.enabled=false` so the plugin's off-heap check fails).
3. Start a Spark shell / submit a job.
4. Observe the driver / executor logs still contain `INFO core/src/lib.rs: Comet native library version ... initialized`.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the initialization path referenced by core/src/lib.rs and the handling of spark.comet.enabled, spark.memory.offHeap.enabled, and spark.comet.exec.onHeap.enabled. Reproduce the disabled-session and failed off-heap-gate cases, then verify that the native-library message no longer claims Comet is initialized when no plan is accelerated.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.