[Bug] Arrow Flight SQL: Iceberg BINARY column is declared as Arrow string but carries invalid UTF-8
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
Apache Doris 4.1.3-rc02, commit `31263df4dc1d4d3a27517d264802cd4d6b92c874`
Client: Python + ADBC Flight SQL driver (`adbc_driver_flightsql`), FE `arrow_flight_sql_port` = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.
### What's Wrong?
When an Iceberg catalog is configured with `"enable.mapping.varbinary" = "false"`, Doris Flight SQL declares an arbitrary `BINARY` column as Arrow `string`, but the payload actually contains non-UTF-8 bytes.
PyArrow fails before returning any row:
```
UnicodeDecodeError (byte 0x84)
```
The Arrow schema for the query is `id: int32, col1: string, col2: string`. JDBC reads the same Iceberg rows without a problem, and with VARBINARY mapping enabled the ADBC binary read works.
### What You Expected?
The Arrow type declared by the server and the encoding of the payload must agree. Arbitrary binary content should be sent as `binary`/`large_binary`, or, if it is sent as `string`, it must be a valid UTF-8 representation.
### How to Reproduce?
1. Bring up the Iceberg regression fixture.
2. Create or use an Iceberg catalog with `"enable.mapping.varbinary" = "false"`.
3. `SWITCH` to that catalog and `USE test_varbinary`.
4. Run the query over Python ADBC, inspect the schema and call `to_pylist()`.
```sql
SWITCH test_iceberg_no_mapping;
USE test_varbinary;
SELECT * FROM test_ice_uuid_orc ORDER BY id;
```
Client side:
```python
import adbc_driver_flightsql.dbapi as flight_sql
conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
cur.execute("SELECT * FROM test_iceberg_no_mapping.test_varbinary.test_ice_uuid_orc ORDER BY id")
table = cur.fetch_arrow_table()
print(table.schema) # id: int32, col1: string, col2: string
print(table.to_pylist()) # UnicodeDecodeError on byte 0x84
```
### Anything Else?
Without VARBINARY mapping, the Iceberg `binary`/`uuid` column falls back to the Doris `STRING` type, and the Arrow conversion follows that declared type rather than the real content. Since Arrow `string` is defined as UTF-8, any consumer that validates the encoding (PyArrow does) fails on the whole batch.
**Workaround:** set `"enable.mapping.varbinary" = "true"` on the Iceberg catalog so the column is returned through the Arrow `binary` type.
Found with the `external_table_p0/iceberg/test_iceberg_varbinary` fixture; four no-mapping cases are affected, while the mapping-enabled binary reads keep passing over ADBC.
Tracking issue: #65615
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by running the external_table_p0/iceberg/test_iceberg_varbinary fixture with enable.mapping.varbinary set to false, then reproduce the query through Python ADBC Flight SQL and inspect its Arrow schema and to_pylist() result. Trace the Iceberg binary/UUID type through the Flight SQL conversion path. Done means the declared Arrow type matches the payload encoding and all four no-mapping cases pass without UnicodeDecodeError, while mapping-enabled reads remain passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, python, sql
- Domain
- api, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100