apache / apache/doris

[Bug] Arrow Flight SQL: UTC TIMESTAMPTZ is tagged with the non-portable Arrow timezone "Z"

Open
#67,370 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

Apache Doris 4.1.3-rc02, commit `31263df4dc1d4d3a27517d264802cd4d6b92c874`

Client: Python + ADBC Flight SQL driver (`adbc_driver_flightsql`), FE `arrow_flight_sql_port` = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.

### What's Wrong?

When reading a mapped Iceberg `TIMESTAMPTZ` column with the session `time_zone` set to `UTC`, Doris Flight SQL returns the Arrow type `timestamp[s, tz=Z]`.

PyArrow treats the timezone string as a `zoneinfo` key, so `to_pylist()` raises `ZoneInfoNotFoundError` because `Z` is not a valid IANA time zone name. The underlying epoch values themselves are correct: `[1735704000, null, 939528000, null]`. Rewriting only the schema metadata from `Z` to `UTC` leaves the epochs unchanged and makes the conversion succeed.

### What You Expected?

UTC timezone metadata should use a name that PyArrow / `zoneinfo` can resolve (e.g. `UTC`, or a fixed `+00:00` offset), so that `to_pylist()` returns UTC datetimes directly.

### How to Reproduce?

1. Bring up the Iceberg regression fixture and create a catalog with `"enable.mapping.timestamp_tz" = "true"`.
2. `SWITCH` to that catalog, `USE test_timestamp_tz`, and set the session `time_zone` to `UTC`.
3. Read `test_ice_timestamp_tz_orc` over Python ADBC Flight SQL.
4. Inspect the Arrow schema and call `to_pylist()`.

```sql
SWITCH test_iceberg_timestamp_tz_with_mapping;
USE test_timestamp_tz;
SET time_zone = 'UTC';
SELECT * FROM test_ice_timestamp_tz_orc ORDER BY id;
```

Client side:

```python
import adbc_driver_flightsql.dbapi as flight_sql

conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
cur.execute("SET time_zone = 'UTC'")
cur.execute("SELECT * FROM test_iceberg_timestamp_tz_with_mapping."
"test_timestamp_tz.test_ice_timestamp_tz_orc ORDER BY id")
table = cur.fetch_arrow_table()
print(table.schema) # timestamp[s, tz=Z]
print(table.to_pylist()) # ZoneInfoNotFoundError
```

### Anything Else?

`Z` is valid ISO-8601 offset syntax but is not a portable Arrow timezone string; the Arrow spec expects either an IANA name or a fixed `+HH:MM` offset. The Doris session `time_zone` value is apparently passed through to the Arrow field metadata verbatim.

**Workaround:** on the client, cast only the Arrow timezone metadata from `Z` to `UTC` before converting to Python; this does not change the epoch values. Casting to `STRING` in SQL also works, but loses the native `TIMESTAMPTZ` Arrow type.

Found with the `external_table_p0/iceberg/test_iceberg_timestamp_tz` fixture and confirmed with a raw Arrow probe plus a metadata-only cast.

Tracking issue: #65615

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Run the external_table_p0/iceberg/test_iceberg_timestamp_tz fixture and reproduce the Arrow Flight SQL result with the session time_zone set to UTC. Trace where the Flight SQL result schema turns the session timezone into Arrow metadata. Done means UTC timestamps use a PyArrow-resolvable timezone such as UTC or +00:00, while epoch values remain unchanged and the regression scenario succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python, sql
Domain
api, database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.