apache / apache/doris

[Bug] Arrow Flight SQL: MAP with a NULL key cannot be returned as an Arrow MAP

Open
#67,372 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

Apache Doris 4.1.3-rc02, commit `31263df4dc1d4d3a27517d264802cd4d6b92c874`

Client: Python + ADBC Flight SQL driver (`adbc_driver_flightsql`), FE `arrow_flight_sql_port` = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.

### What's Wrong?

Doris `MAP` semantics allow a `NULL` key, but the Arrow `MAP` specification requires the key field to be non-nullable. When a result contains a `NULL` map key, the Flight SQL `DoGet` fails and the whole result becomes unreadable:

```
Can not write null value of map key to arrow
```

Projecting the same value as `map_entries(m)`, i.e. `ARRAY>`, is returned over ADBC correctly and preserves the `NULL` key.

### What You Expected?

Flight SQL should be able to return `MAP` data that Doris itself accepts, either through a compatible Arrow representation that preserves the `NULL` key, or with a clear, documented limitation; it should not fail the entire result at `DoGet` time.

### How to Reproduce?

1. Create a Doris table with a `MAP` column.
2. Insert `map(NULL,100)`.
3. Query the raw `MAP` column over Python ADBC; `DoGet` fails.
4. Query `map_entries(m)`; the same data is returned through the compatible structure.

```sql
DROP TABLE IF EXISTS adbc_null_map;
CREATE TABLE adbc_null_map (
k INT,
m MAP
) DUPLICATE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES("replication_num"="1");

INSERT INTO adbc_null_map VALUES (1, map(NULL,100));

-- Fails through Arrow Flight SQL.
SELECT m FROM adbc_null_map;

-- Succeeds and preserves the NULL key.
SELECT map_entries(m) FROM adbc_null_map;
```

Client side:

```python
import adbc_driver_flightsql.dbapi as flight_sql

conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
cur.execute("SELECT m FROM adbc_null_map")
cur.fetch_arrow_table()
```

### Anything Else?

This is a genuine model mismatch between Doris `MAP` and Arrow `MAP`, so it needs an explicit decision rather than only a test skip: either serialize such maps as `list>` (which round-trips the `NULL` key, as `map_entries` already shows), or reject them with a clear, documented error at plan/schema time instead of failing mid-stream in `DoGet`. Either way the limitation belongs in the Arrow Flight SQL documentation.

Related: #65182 skipped the Arrow-incompatible Map null-key regression cases; this issue tracks the underlying behavior.

**Workaround:** use `map_entries(m)` in SQL to project the `MAP` as `ARRAY>`, or filter/replace `NULL` keys when the business logic allows it.

Tracking issue: #65615

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with the provided CREATE TABLE, INSERT, SELECT, and Python ADBC Flight SQL calls, then compare raw MAP with map_entries(m) through DoGet. Read the Arrow Flight SQL serialization path and related null-key regression cases from #65182; done means choosing and documenting a compatible representation or an explicit schema-time error instead of a mid-stream failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python, sql
Domain
api, backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.