apache / apache/datafusion-comet

Add metrics and logging to JVM-scalar-UDF dispatch path

Open
#4,176 0 comments 0 reactions 0 assignees View on GitHub
area:expressions priority:low
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Describe the problem

The JVM-scalar-UDF dispatch path (on the prototype branch `prototype-jvm-scalar-udf`) is invisible at runtime:

- No SQL metric distinguishes "rlike ran on the JVM path" from "rlike ran natively".
- No log line on first dispatch per executor identifies which UDF is being routed and which classloader resolved it.
- `EXPLAIN` output for `JvmScalarUdfExpr` is not surfaced in Spark's plan summary.

This makes "why is rlike slow in prod?" or "why is the JVM path being chosen?" unanswerable without source-level debugging.

## Describe the potential solution

Add SQL metrics on the parent `CometNativeExec` (or a dedicated metric collector keyed by UDF class name):

- `numJvmUdfBatches` — counter, batches dispatched
- `jvmUdfElapsedNanos` — accumulator, time spent in JNI + UDF body
- `jvmUdfBytesIn` / `jvmUdfBytesOut` — optional, for transfer-cost visibility

And add an `info`-level log per `(executor, UDF class name)` on first dispatch, naming the resolved class and the classloader that resolved it.

Optionally, surface `JvmScalarUdf()` as the operator name in EXPLAIN by hooking the appropriate Comet plan-formatter site.

## Additional context

Identified during code review of the JVM-scalar-UDF prototype. Filed as a follow-up so the prototype PR can ship without a metrics-design decision.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.