[Bug] Arrow Flight SQL: UTC TIMESTAMPTZ is tagged with the non-portable Arrow timezone "Z"
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
Apache Doris 4.1.3-rc02, commit `31263df4dc1d4d3a27517d264802cd4d6b92c874`
Client: Python + ADBC Flight SQL driver (`adbc_driver_flightsql`), FE `arrow_flight_sql_port` = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.
### What's Wrong?
When reading a mapped Iceberg `TIMESTAMPTZ` column with the session `time_zone` set to `UTC`, Doris Flight SQL returns the Arrow type `timestamp[s, tz=Z]`.
PyArrow treats the timezone string as a `zoneinfo` key, so `to_pylist()` raises `ZoneInfoNotFoundError` because `Z` is not a valid IANA time zone name. The underlying epoch values themselves are correct: `[1735704000, null, 939528000, null]`. Rewriting only the schema metadata from `Z` to `UTC` leaves the epochs unchanged and makes the conversion succeed.
### What You Expected?
UTC timezone metadata should use a name that PyArrow / `zoneinfo` can resolve (e.g. `UTC`, or a fixed `+00:00` offset), so that `to_pylist()` returns UTC datetimes directly.
### How to Reproduce?
1. Bring up the Iceberg regression fixture and create a catalog with `"enable.mapping.timestamp_tz" = "true"`.
2. `SWITCH` to that catalog, `USE test_timestamp_tz`, and set the session `time_zone` to `UTC`.
3. Read `test_ice_timestamp_tz_orc` over Python ADBC Flight SQL.
4. Inspect the Arrow schema and call `to_pylist()`.
```sql
SWITCH test_iceberg_timestamp_tz_with_mapping;
USE test_timestamp_tz;
SET time_zone = 'UTC';
SELECT * FROM test_ice_timestamp_tz_orc ORDER BY id;
```
Client side:
```python
import adbc_driver_flightsql.dbapi as flight_sql
conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
cur.execute("SET time_zone = 'UTC'")
cur.execute("SELECT * FROM test_iceberg_timestamp_tz_with_mapping."
"test_timestamp_tz.test_ice_timestamp_tz_orc ORDER BY id")
table = cur.fetch_arrow_table()
print(table.schema) # timestamp[s, tz=Z]
print(table.to_pylist()) # ZoneInfoNotFoundError
```
### Anything Else?
`Z` is valid ISO-8601 offset syntax but is not a portable Arrow timezone string; the Arrow spec expects either an IANA name or a fixed `+HH:MM` offset. The Doris session `time_zone` value is apparently passed through to the Arrow field metadata verbatim.
**Workaround:** on the client, cast only the Arrow timezone metadata from `Z` to `UTC` before converting to Python; this does not change the epoch values. Casting to `STRING` in SQL also works, but loses the native `TIMESTAMPTZ` Arrow type.
Found with the `external_table_p0/iceberg/test_iceberg_timestamp_tz` fixture and confirmed with a raw Arrow probe plus a metadata-only cast.
Tracking issue: #65615
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Run the external_table_p0/iceberg/test_iceberg_timestamp_tz fixture and reproduce the Arrow Flight SQL result with the session time_zone set to UTC. Trace where the Flight SQL result schema turns the session timezone into Arrow metadata. Done means UTC timestamps use a PyArrow-resolvable timezone such as UTC or +00:00, while epoch values remain unchanged and the regression scenario succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, sql
- Domain
- api, database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100