apache / apache/datafusion-comet

unbase64 can fail on rows skipped by LIMIT and semi/anti joins

Open
#5,532 0 comments 0 reactions 0 assignees View on GitHub
area:expressions bug priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### Describe the bug

Comet can fail while decoding malformed Base64 on rows that Spark never evaluates. A limit may stop after the first row, or a semi/anti join may stop checking candidates after finding a match. Both native `unbase64` and the JVM codegen dispatcher can evaluate a whole batch before that early termination takes effect.

This means enabling Comet can turn a successful Spark query into `Last unit does not have enough valid bits`, even when the malformed value does not contribute to the result.

### Steps to reproduce

Start Spark with the Comet extension, native execution, and the Comet shuffle manager configured. Create the input with Comet disabled, then compare the same query with Comet disabled and enabled:

```sql
SET spark.comet.enabled = false;
SET spark.sql.shuffle.partitions = 1;

CREATE TABLE unbase64_limit_input USING parquet AS
SELECT /*+ COALESCE(1) */ bad
FROM VALUES ('YWJj'), ('A') AS v(bad);

SELECT hex(unbase64(bad)) FROM unbase64_limit_input LIMIT 1;

SET spark.comet.enabled = true;

SELECT hex(unbase64(bad)) FROM unbase64_limit_input LIMIT 1;
```

In the reproduced single-file scan order, `YWJj` is read before `A`. Spark returns `616263` (the bytes for `abc`). Comet instead throws on the second value. This example relies on that physical input order; it does not assume that SQL without `ORDER BY` guarantees ordering.

The failing Comet plan is:

```text
CometCollectLimit 1
+- CometProject [hex(unbase64(bad))]
+- CometNativeScan parquet
```

The same problem occurs when the decoder is in a filter below `LIMIT`, in semi/anti join residual conditions, or below an already ordered `WindowGroupLimit`. Compound inputs and `to_binary(bad, 'base64')` can reach the JVM dispatcher and still fail for the same reason.

### Expected behavior

Match Spark's evaluation behavior: preserve successful queries when Spark skips the malformed value, while still raising an error when Spark actually decodes it. Ordinary native decoding should remain available when early termination cannot skip the value.

### Additional context

Reproduced on Apache Comet `78defce020ac3c4d2b32ff7f172f3018b7587a48` with Spark 4.1.3, a fresh native/JVM build, and verified Comet physical operators. The relevant implementation is unchanged at `a223ba14e9f1445b5d315962ae9005f7ec45b187`.

Seven query shapes produced 28 Spark-versus-Comet mismatches across all four combinations of ANSI mode and AQE. Sixteen comparisons using valid inputs or deliberately consumed malformed values matched Spark. This is not specific to ANSI mode, and switching to JVM codegen dispatch alone does not preserve row-by-row evaluation.

Contributor guide

Open the contributing guide

Research direction

Start with the CometCollectLimit, CometProject, and CometNativeScan plan in the reproduction, then trace native unbase64 and the JVM codegen dispatcher through the relevant implementation at the cited commit. Run the supplied LIMIT query and the semi/anti join, window, and to_binary cases against Spark and Comet. Done means skipped malformed values no longer fail, while values Spark actually decodes still raise the error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala, sql
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.