apache / apache/doris

[Bug] Arrow Flight SQL: Iceberg BINARY column is declared as Arrow string but carries invalid UTF-8

Open
#67,371 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

Apache Doris 4.1.3-rc02, commit `31263df4dc1d4d3a27517d264802cd4d6b92c874`

Client: Python + ADBC Flight SQL driver (`adbc_driver_flightsql`), FE `arrow_flight_sql_port` = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.

### What's Wrong?

When an Iceberg catalog is configured with `"enable.mapping.varbinary" = "false"`, Doris Flight SQL declares an arbitrary `BINARY` column as Arrow `string`, but the payload actually contains non-UTF-8 bytes.

PyArrow fails before returning any row:

```
UnicodeDecodeError (byte 0x84)
```

The Arrow schema for the query is `id: int32, col1: string, col2: string`. JDBC reads the same Iceberg rows without a problem, and with VARBINARY mapping enabled the ADBC binary read works.

### What You Expected?

The Arrow type declared by the server and the encoding of the payload must agree. Arbitrary binary content should be sent as `binary`/`large_binary`, or, if it is sent as `string`, it must be a valid UTF-8 representation.

### How to Reproduce?

1. Bring up the Iceberg regression fixture.
2. Create or use an Iceberg catalog with `"enable.mapping.varbinary" = "false"`.
3. `SWITCH` to that catalog and `USE test_varbinary`.
4. Run the query over Python ADBC, inspect the schema and call `to_pylist()`.

```sql
SWITCH test_iceberg_no_mapping;
USE test_varbinary;
SELECT * FROM test_ice_uuid_orc ORDER BY id;
```

Client side:

```python
import adbc_driver_flightsql.dbapi as flight_sql

conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
cur.execute("SELECT * FROM test_iceberg_no_mapping.test_varbinary.test_ice_uuid_orc ORDER BY id")
table = cur.fetch_arrow_table()
print(table.schema) # id: int32, col1: string, col2: string
print(table.to_pylist()) # UnicodeDecodeError on byte 0x84
```

### Anything Else?

Without VARBINARY mapping, the Iceberg `binary`/`uuid` column falls back to the Doris `STRING` type, and the Arrow conversion follows that declared type rather than the real content. Since Arrow `string` is defined as UTF-8, any consumer that validates the encoding (PyArrow does) fails on the whole batch.

**Workaround:** set `"enable.mapping.varbinary" = "true"` on the Iceberg catalog so the column is returned through the Arrow `binary` type.

Found with the `external_table_p0/iceberg/test_iceberg_varbinary` fixture; four no-mapping cases are affected, while the mapping-enabled binary reads keep passing over ADBC.

Tracking issue: #65615

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by running the external_table_p0/iceberg/test_iceberg_varbinary fixture with enable.mapping.varbinary set to false, then reproduce the query through Python ADBC Flight SQL and inspect its Arrow schema and to_pylist() result. Trace the Iceberg binary/UUID type through the Flight SQL conversion path. Done means the declared Arrow type matches the payload encoding and all four no-mapping cases pass without UnicodeDecodeError, while mapping-enabled reads remain passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, python, sql
Domain
api, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.