apache / apache/doris

[Bug] Arrow Flight SQL: first VARIANT read on a pre-existing connection returns an empty string instead of {}

Open
#67,367 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

Apache Doris 4.1.3-rc02, commit `31263df4dc1d4d3a27517d264802cd4d6b92c874`

Client: Python + ADBC Flight SQL driver (`adbc_driver_flightsql`), FE `arrow_flight_sql_port` = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.

### What's Wrong?

If an ADBC Flight SQL connection is opened *before* a `VARIANT` table is created and written, the first read of a persisted empty JSON object on that connection can return an empty string instead of `{}`.

Running exactly the same query a second time on the same connection returns `{}`, and a freshly opened ADBC connection also returns `{}` on its first try.

The raw Arrow schema is `int32, string, string`:

- pre-existing connection, first read: `["1", "", ""]`
- same connection, second read: `["1", "{}", "{}"]`
- new connection after the write: `["1", "{}", "{}"]`

### What You Expected?

The first read and all subsequent reads should be identical. An empty JSON object should come back as `{}` for both nullable and `NOT NULL` `VARIANT` columns.

### How to Reproduce?

1. Open the ADBC Flight SQL connection first, without querying the target table yet.
2. Over a separate MySQL/JDBC connection, run the DDL and `INSERT` below.
3. On the pre-existing ADBC connection, run the same `SELECT` twice in a row.
4. Compare the two Arrow results, then open a new ADBC connection and run it once more as a control.

```sql
DROP TABLE IF EXISTS adbc_variant_first_read;
CREATE TABLE adbc_variant_first_read (
id INT,
v_nullable VARIANT NULL,
v_not_null VARIANT NOT NULL
) DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES("replication_num"="1");

INSERT INTO adbc_variant_first_read
VALUES (1, PARSE_JSON('{}'), PARSE_JSON('{}'));

SELECT id, CAST(v_nullable AS STRING), CAST(v_not_null AS STRING)
FROM adbc_variant_first_read ORDER BY id;
```

Client side:

```python
import adbc_driver_flightsql.dbapi as flight_sql

# Open this connection BEFORE the JDBC session runs CREATE TABLE and INSERT.
conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
sql = ("SELECT id, CAST(v_nullable AS STRING), CAST(v_not_null AS STRING) "
"FROM adbc_variant_first_read ORDER BY id")
cur.execute(sql)
print(cur.fetch_arrow_table().to_pylist()) # first read may contain empty strings
cur.execute(sql)
print(cur.fetch_arrow_table().to_pylist()) # returns {}
```

### Anything Else?

This looks like stale per-connection schema/metadata state on the Flight SQL session: the connection was established before the table existed, and the first read resolves the variant subcolumns against that stale state.

**Workaround:** retry the identical read, or re-establish the ADBC connection after the write completes. A silent retry was deliberately not added to the test path, because it would hide the consistency problem.

Also reproducible through the `variant_p0/variant_hirachinal` regression case, in addition to the isolated table above.

Tracking issue: #65615

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with the isolated ADBC Flight SQL reproduction and compare the first and second Arrow results for nullable and NOT NULL VARIANT columns. Then inspect the Flight SQL session's per-connection schema or metadata handling, using the mentioned variant_p0/variant_hirachinal regression case as a reference. Done means the first read on a pre-existing connection matches later reads and a fresh connection, returning {} for both columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.