apache / apache/datafusion-comet
unbase64 can fail on rows skipped by LIMIT and semi/anti joins
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### Describe the bug
Comet can fail while decoding malformed Base64 on rows that Spark never evaluates. A limit may stop after the first row, or a semi/anti join may stop checking candidates after finding a match. Both native `unbase64` and the JVM codegen dispatcher can evaluate a whole batch before that early termination takes effect.
This means enabling Comet can turn a successful Spark query into `Last unit does not have enough valid bits`, even when the malformed value does not contribute to the result.
### Steps to reproduce
Start Spark with the Comet extension, native execution, and the Comet shuffle manager configured. Create the input with Comet disabled, then compare the same query with Comet disabled and enabled:
```sql
SET spark.comet.enabled = false;
SET spark.sql.shuffle.partitions = 1;
CREATE TABLE unbase64_limit_input USING parquet AS
SELECT /*+ COALESCE(1) */ bad
FROM VALUES ('YWJj'), ('A') AS v(bad);
SELECT hex(unbase64(bad)) FROM unbase64_limit_input LIMIT 1;
SET spark.comet.enabled = true;
SELECT hex(unbase64(bad)) FROM unbase64_limit_input LIMIT 1;
```
In the reproduced single-file scan order, `YWJj` is read before `A`. Spark returns `616263` (the bytes for `abc`). Comet instead throws on the second value. This example relies on that physical input order; it does not assume that SQL without `ORDER BY` guarantees ordering.
The failing Comet plan is:
```text
CometCollectLimit 1
+- CometProject [hex(unbase64(bad))]
+- CometNativeScan parquet
```
The same problem occurs when the decoder is in a filter below `LIMIT`, in semi/anti join residual conditions, or below an already ordered `WindowGroupLimit`. Compound inputs and `to_binary(bad, 'base64')` can reach the JVM dispatcher and still fail for the same reason.
### Expected behavior
Match Spark's evaluation behavior: preserve successful queries when Spark skips the malformed value, while still raising an error when Spark actually decodes it. Ordinary native decoding should remain available when early termination cannot skip the value.
### Additional context
Reproduced on Apache Comet `78defce020ac3c4d2b32ff7f172f3018b7587a48` with Spark 4.1.3, a fresh native/JVM build, and verified Comet physical operators. The relevant implementation is unchanged at `a223ba14e9f1445b5d315962ae9005f7ec45b187`.
Seven query shapes produced 28 Spark-versus-Comet mismatches across all four combinations of ANSI mode and AQE. Sixteen comparisons using valid inputs or deliberately consumed malformed values matched Spark. This is not specific to ANSI mode, and switching to JVM codegen dispatch alone does not preserve row-by-row evaluation.
Contributor guide
Research direction
Start with the CometCollectLimit, CometProject, and CometNativeScan plan in the reproduction, then trace native unbase64 and the JVM codegen dispatcher through the relevant implementation at the cited commit. Run the supplied LIMIT query and the semi/anti join, window, and to_binary cases against Spark and Comet. Done means skipped malformed values no longer fail, while values Spark actually decodes still raise the error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala, sql
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100