[Bug] Arrow Flight SQL: MAP with a NULL key cannot be returned as an Arrow MAP
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
Apache Doris 4.1.3-rc02, commit `31263df4dc1d4d3a27517d264802cd4d6b92c874`
Client: Python + ADBC Flight SQL driver (`adbc_driver_flightsql`), FE `arrow_flight_sql_port` = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.
### What's Wrong?
Doris `MAP` semantics allow a `NULL` key, but the Arrow `MAP` specification requires the key field to be non-nullable. When a result contains a `NULL` map key, the Flight SQL `DoGet` fails and the whole result becomes unreadable:
```
Can not write null value of map key to arrow
```
Projecting the same value as `map_entries(m)`, i.e. `ARRAY>`, is returned over ADBC correctly and preserves the `NULL` key.
### What You Expected?
Flight SQL should be able to return `MAP` data that Doris itself accepts, either through a compatible Arrow representation that preserves the `NULL` key, or with a clear, documented limitation; it should not fail the entire result at `DoGet` time.
### How to Reproduce?
1. Create a Doris table with a `MAP` column.
2. Insert `map(NULL,100)`.
3. Query the raw `MAP` column over Python ADBC; `DoGet` fails.
4. Query `map_entries(m)`; the same data is returned through the compatible structure.
```sql
DROP TABLE IF EXISTS adbc_null_map;
CREATE TABLE adbc_null_map (
k INT,
m MAP
) DUPLICATE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES("replication_num"="1");
INSERT INTO adbc_null_map VALUES (1, map(NULL,100));
-- Fails through Arrow Flight SQL.
SELECT m FROM adbc_null_map;
-- Succeeds and preserves the NULL key.
SELECT map_entries(m) FROM adbc_null_map;
```
Client side:
```python
import adbc_driver_flightsql.dbapi as flight_sql
conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
cur.execute("SELECT m FROM adbc_null_map")
cur.fetch_arrow_table()
```
### Anything Else?
This is a genuine model mismatch between Doris `MAP` and Arrow `MAP`, so it needs an explicit decision rather than only a test skip: either serialize such maps as `list>` (which round-trips the `NULL` key, as `map_entries` already shows), or reject them with a clear, documented error at plan/schema time instead of failing mid-stream in `DoGet`. Either way the limitation belongs in the Arrow Flight SQL documentation.
Related: #65182 skipped the Arrow-incompatible Map null-key regression cases; this issue tracks the underlying behavior.
**Workaround:** use `map_entries(m)` in SQL to project the `MAP` as `ARRAY>`, or filter/replace `NULL` keys when the business logic allows it.
Tracking issue: #65615
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by reproducing the failure with the provided CREATE TABLE, INSERT, SELECT, and Python ADBC Flight SQL calls, then compare raw MAP with map_entries(m) through DoGet. Read the Arrow Flight SQL serialization path and related null-key regression cases from #65182; done means choosing and documenting a compatible representation or an explicit schema-time error instead of a mid-stream failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, sql
- Domain
- api, backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100