apache / apache/datafusion-comet

Comet native scan rejects invalid UTF-8 byte sequences in STRING column (hll.sql on Spark 4.1)

Open
#4,121 1 comment 0 reactions 0 assignees View on GitHub
area:scan priority:medium spark 4.1
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

## Describe the bug

Spark allows storing arbitrary byte sequences in `STRING` columns, including bytes that are not valid UTF-8 (for example via `CAST(X'C1' AS STRING)` and `CAST(X'80' AS STRING)`). Comet's native Parquet scan rejects these rows with:

```
org.apache.comet.CometNativeException
Arrow: Parquet argument error: Parquet error: encountered non UTF-8 data
```

This surfaces in Spark 4.1.1's `hll.sql` (newly added in 4.1) at query #10:

```sql
SELECT hll_sketch_estimate(hll_sketch_agg(s)) utf8_b FROM hll_string_test;
```

where `hll_string_test` is populated with `INSERT INTO hll_string_test VALUES (''), (' '), (CAST(X'C1' AS STRING)), (CAST(X'80' AS STRING)), ...` to exercise different collations including invalid Unicode bytes.

## Steps to reproduce

Run Spark 4.1.1's SQL test suite with Comet enabled (the `Spark SQL Tests` matrix entry for 4.1.1). The test `hll.sql` in `SQLQueryTestSuite` fails.

## Expected behavior

Comet should either (a) accept invalid-UTF-8 bytes in STRING columns the way Spark does, or (b) fall back to Spark when reading STRING columns whose Parquet data contains invalid UTF-8.

## Workaround

`hll.sql` is currently disabled when Comet is enabled via `--SET spark.comet.enabled = false` at the top of the file in `dev/diffs/4.1.1.diff` (introduced as part of the Spark 4.1.1 SQL test matrix landing).

## Additional context

PR #4093 enables Spark 4.1.1 in the `Spark SQL Tests` workflow.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.