[Tracking] Arrow Flight SQL and ADBC Catalog: issues and improvements

This issue tracks Arrow Flight SQL-related problems in Apache Doris. If you encounter a problem, please open a new issue or leave a comment here.

## Goal

Use this issue as the single index for user-visible bugs, protocol gaps, reliability work, and ecosystem improvements related to Doris Arrow Flight SQL and the ADBC or Flight SQL clients that use it.

ADBC is the client API and driver layer. Doris implements the Arrow Flight SQL server path that those clients call. This tracker covers both sides when a client report identifies a Doris interoperability or server behavior problem.

Foundational context: #25514 and #19812.

### Status convention

- [ ] Open, unmerged, closed without a verified fix, duplicated but not closed, or otherwise needs follow-up.
- [x] The referenced change was merged or the work item was confirmed complete. Backports are listed beside their primary PR.

## Open or needs follow-up

### Reliability, lifecycle, and correctness

- [x] **External-table split lifecycle across `GetFlightInfo` and `DoGet`:** #62259 is closed and the root-cause fix #64799 merged on master on 2026-07-18, so only release-branch coverage is missing: there is no branch-4.1 or branch-4.0 backport of #64799. #64797 and its 4.1 backport #64890 harden the error path so the BE does not crash; they explicitly do not keep the `SplitSource` alive. Check this item once #64799 reaches the supported branches.
- [ ] **Corrupted large string/JSON results:** #61124 reports invalid StringArray offsets and an empty data buffer.
- [ ] **DDL regression on 4.0.5:** #62017. Proposed fix #62018 was closed without merge. Reproduce on supported 4.0/4.1 builds and land or replace the fix.
- [ ] **BE crash with `public_host` and an ILB proxy:** #62217. Separate the endpoint-routing problem from the Arrow/gRPC status-conversion crash path and verify both.
- [ ] **LargeUtf8 offset handling can crash remote Doris reads:** the only proposed fix, PR #62681, was closed without merge on 2026-07-22, and no issue tracks the crash itself. Reproduce on a supported build and file a report, or land a replacement fix.
- [ ] **Complex-type fidelity:** #64976 proposed preserving array child nullability and #65044 added focused IPC tests, but both closed without merge. #65182 documents additional Arrow Map null-key and raw BITMAP incompatibilities by skipping those regression cases. Decide which cases require engine fixes and which require documented protocol limitations.
- [ ] **Flight SQL JDBC `RST_STREAM` failures on 4.0.1:** #59304 was closed as stale without a linked fix. Reproduce against current supported versions or link the resolving change.
- [ ] **Flight SQL with the distributed planner disabled:** #57318 was closed as stale. #63136 appears to address the same empty-result-receiver failure on master and #63155 backports it to 4.1, but the 4.0 backport #63154 closed without merge. Verify the version matrix and close the loop explicitly.
- [ ] **Sessions remain in `SHOW PROCESSLIST` after an ADBC connection closes:** #36331 was closed as stale. #55008/#55396 improve timeout cleanup and #65040 keeps the timeout checker alive (4.1 backport #65063 merged, 4.0 backport #65062 still open), but explicit client-close behavior still needs a current-version verification.

### Arrow type fidelity and client-visible behavior

Reported from an end-to-end Python ADBC sweep against 4.1.3-rc02 (commit `31263df4dc`), with MySQL/JDBC used as the control path. Each item lists a workaround, so they are usable as documented limitations until they are fixed.

- [ ] **DATETIME was returned as a timezone-aware Arrow timestamp:** #65741. The naive-DATETIME work listed below appears to fix this on master and branch-4.1, but no PR declared that it closes the report. Verify on a current build and close it, or record what still reproduces.
- [ ] **Raw `HLL` and `QUANTILE_STATE` reads fail with `INTERNAL`:** #67364. `StmtExecutor.sendFields()` fails a `Preconditions.checkState` instead of returning a clean unsupported-type error. Same family as the raw `BITMAP` case skipped in #65182.
- [ ] **Year-zero `DATE` values are off by one day:** #67366. `0000-01-01` arrives as `0000-01-02` over Arrow while JDBC is correct; looks like a proleptic-Gregorian versus Julian mismatch in the `date32` day computation.
- [ ] **First `VARIANT` read on a pre-existing connection loses empty objects:** #67367. A connection opened before the table was created returns `""` instead of `{}` on the first read only; the second read on the same connection is correct.
- [ ] **Short-circuit point query returns no Flight endpoint:** #67368. `fetch arrow flight schema failed, no FlightSqlEndpointsLocations`; the same query succeeds with `enable_short_circuit_query=false`, so the short-circuit path appears not to register a Flight endpoint at all.
- [ ] **PL/SQL `CALL` errors but executes its DML side effect four times:** #67369. Returns `getMysqlChannel not in mysql connection` and still inserts four rows for a single call. Data-correctness impact; clients must not auto-retry on this error.
- [ ] **UTC `TIMESTAMPTZ` uses the non-portable Arrow timezone `Z`:** #67370. PyArrow resolves the timezone string through `zoneinfo` and raises `ZoneInfoNotFoundError`; the epoch values themselves are correct.
- [ ] **Iceberg `BINARY` is declared as Arrow `string` with a non-UTF-8 payload:** #67371. With `enable.mapping.varbinary=false` the declared type and the payload encoding disagree, so PyArrow fails the whole batch on `UnicodeDecodeError`.
- [ ] **Doris `MAP` with a `NULL` key cannot be returned as an Arrow `MAP`:** #67372. `DoGet` fails with `Can not write null value of map key to arrow`; `map_entries()` round-trips the same value, which suggests `list>` as one possible representation. Complements the skip added in #65182.

### Authentication, endpoint advertisement, and session semantics

- [ ] **Duplicate authentication reports:** #61743, #61744, and #61757 all report authentication using `user@0.0.0.0`. The fix #63506 and both backports #63591/#63592 are merged, but all three reports are still open. Verify the merged fix against the reported read-only user case, then close or deduplicate them.
- [ ] **FE cannot advertise a public Flight endpoint:** #62538 requests FE equivalents of BE `public_host` and the Flight proxy port.
- [ ] **FE/BE advertises `127.0.0.1:8050` to a remote JDBC client:** #59490 was closed as stale. Check whether it is the same endpoint-advertisement gap as #62538.
- [ ] **Kubernetes and reverse-proxy topology:** #44599 is an older, stale-closed precursor to #62217/#62538. Decide whether the remaining work belongs in Doris, Doris Operator, or both, and cross-link the owner issue.
- [ ] **Doris Catalog session-variable passthrough in Arrow Flight mode:** #59742.

### Protocol coverage and architecture

- [ ] **Protocol-agnostic session and execution layer:** #67577 tracks making MySQL and Arrow Flight SQL equal front ends over one session layer - one result path, one Doris-to-Arrow type mapping, one connection pool and quota, one prepared-statement registry. Its stage 0 bug fixes are already up (#67569, #67572, #67576), and its stage 2 (a single, table-tested type mapping) is where several type-fidelity items in this tracker would be settled rather than fixed one by one: #67364, #67370, #67371 and #67372.
- [ ] **Complete Flight SQL server and an official ADBC driver:** #67578 tracks bringing the server up to the spec's feature set - session actions, parallel endpoints, native metadata commands, `CommandStatementIngest` ingestion, cancellation/renewal/polling, IPC compression - and maintaining a `doris` driver in the ADBC Driver Foundry. Several items in this tracker fall inside its scope: endpoint advertisement and reachability (#62538, #59490, #62217, #44599), session-variable passthrough (#59742), session cleanup on client close (#36331), prepared-statement parameter binding ([apache/arrow-java#1009](https://github.com/apache/arrow-java/issues/1009)), the initial database in a connection URI ([apache/arrow-java#829](https://github.com/apache/arrow-java/issues/829)), and ADBC ingestion (#25514).
- [ ] **Variant over cross-cluster Arrow Flight:** #61883. The discussion proposes a LargeUtf8 first step and leaves a more efficient representation as follow-up.
- [ ] **Prepared-statement parameter binding:** [apache/arrow-java#1009](https://github.com/apache/arrow-java/issues/1009) was closed upstream as a duplicate, with the gap attributed to the Doris server side. Create or link a Doris issue that defines the required Flight SQL parameter schema and execution behavior.
- [ ] **Initial database/schema in a Flight SQL JDBC URL:** [apache/arrow-java#829](https://github.com/apache/arrow-java/issues/829) remains open. Determine client-versus-server ownership and document the supported connection option.
- [ ] **ADBC ingestion:** users requested `adbc_ingest`/DataFrame writes in #25514, but no dedicated Doris work item tracks scope, protocol methods, atomicity, or type coverage. Create one if ingestion remains on the roadmap.
- [ ] **Protocol SPI extraction:** #60355.
- [ ] **Optional BE Arrow Flight worker pool:** #60388 adds an `enable_arrow_flight` switch so unused deployments do not allocate the pool.

### Tests and ecosystem

- [ ] **Arrow regression cleanup:** #62536 remains open.
- [ ] **Arrow JDBC close-time allocator leak:** #64024 changed the authentication regression test to avoid the prepared path, and #64677 explicitly says the intermittent close-time leak needs a separate fix. Create or link that follow-up and check whether newer Arrow/Netty versions resolve it.
- [ ] **Spark column projection:** [apache/doris-spark-connector#333](https://github.com/apache/doris-spark-connector/issues/333) reports that `doris.read.fields` is ignored in Arrow read mode.
- [ ] **Spark array work cleanup:** [apache/doris-spark-connector#345](https://github.com/apache/doris-spark-connector/pull/345) remains open although the narrower opt-in implementation [#362](https://github.com/apache/doris-spark-connector/pull/362) merged. Mark the older PR superseded or document the remaining element-type work.

## Completed or merged during the review window

### Core feature and query path

- [x] Doris Catalog and cross-cluster query over Arrow Flight: #55274 and #57898.
- [x] Connection cleanup and thread-local correctness: #55008 with 3.1 backport #55396, plus #55809.
- [x] Non-destructive Flight ticket/query-id parsing: #57240 with 4.0 backport #57330.
- [x] Flight processor close ordering and noisy warning fix: #57858 with 4.0 backport #58011, closing #57859.
- [x] Queries containing `&` no longer produce malformed tickets: #58830 with 4.0 backport #58884.
- [x] Async result completion avoids dirty reads: #60282 with 4.0 backport #60325.
- [x] Remote Flight SQL result receiver initialization: #63136 with 4.1 backport #63155.
- [x] Arrow UTF8/String size limit corrected to the intended 2 GiB threshold: #63137 with backports #63148 and #63558.
- [x] `CommandGetTables` schema and the TIMESTAMPTZ Arrow reader corrected: #66344 with 4.1 backport #66355. The FE `GetTables` mapping had drifted from what BE actually writes.
- [x] DATETIME is returned as a timezone-naive Arrow timestamp so clients stop reading a wall-clock value as an instant: #66873 with 4.1 backport #67027, plus the Thrift path in #67232 with 4.1 backport #67299. The earlier attempt #65823 was closed without merge. This is the likely fix for #65741, which is still open.
- [x] ADBC catalog that reads an external source over Arrow: #66331. Adds an `adbc` catalog type so rows arrive as Arrow record batches without per-value JNI conversion and a scan is split across backends. Master only; no backport.
- [x] Timeout checking continues after one connection throws: #65040 with 4.1 backport #65063. The 4.0 backport #65062 is still open.

### Authentication, diagnostics, and crash hardening

- [x] Better malformed-ticket logging: #58811 with 4.0 backport #58842.
- [x] Workload-group values are populated in audit records for early query failures: #62651 with 4.1 backport #62697.
- [x] Arrow status initialization crash fix: #63191 with backports #63266/#63267. The build-mode root cause was also addressed by #64390 and documented upstream in [apache/arrow#49781](https://github.com/apache/arrow/issues/49781).
- [x] Real client IP is used for Flight authentication: #63506 with backports #63591/#63592.
- [x] Bearer tokens are no longer written to `fe.log` in cleartext: #66572 with 4.1 backport #67146. A token is a complete credential for `arrow_flight_token_alive_time_second` (86400s by default), so any deployment shipping FE logs off-box wants this; there is no 4.0 backport yet.
- [x] External split-fetch failures now fail gracefully instead of crashing the BE: #64797 with 4.1 backport #64890. This is only the robustness half of #62259; the lifecycle item remains open above.
- [x] FE direct-memory leak during prepared-statement creation fixed: #65311 (closes #65305) closes the temporary `VectorSchemaRoot`s in `createPreparedStatement` and adds a repeat-prepare regression test asserting the Flight SQL channel allocator returns to zero. The earlier attempt #52245 was closed without merge. Backports have since merged: 4.1 #65751 and 4.0 #66431.

### Regression framework and ecosystem

- [x] Remote-IP auth regression uses a stable JDBC statement path: #64024 with backports #64047/#64049.
- [x] Regression framework Arrow/Netty versions aligned with FE and JDK 17: #64677.
- [x] JDBC complex values are materialized before result-set teardown: #65084.
- [x] Regression cases now account for Arrow-incompatible Map null keys and protocol-specific BITMAP bytes: #65182.
- [x] Spark connector FE/BE load balancing: [apache/doris-spark-connector#329](https://github.com/apache/doris-spark-connector/pull/329).
- [x] Spark connector Flight read logging: [apache/doris-spark-connector#354](https://github.com/apache/doris-spark-connector/pull/354).
- [x] Spark connector opt-in native array reads: [apache/doris-spark-connector#362](https://github.com/apache/doris-spark-connector/pull/362), closing [#341](https://github.com/apache/doris-spark-connector/issues/341).
- [x] Flink connector BackendClient typo that could affect Flight reads: [apache/doris-flink-connector#622](https://github.com/apache/doris-flink-connector/pull/622).
- [x] Website corrected the Flink connector `source.use-flight-sql` default for current/4.x docs: [apache/doris-website#3725](https://github.com/apache/doris-website/pull/3725).

## Closed without merge or superseded attempts

Keep these for history so the same investigation is not repeated:

- Prepared-statement resource release: #52245.
- Query profile and connection finalization: #45416.
- Early Arrow regression experiments: #53284, #53285, and #53286.
- Alternative generic Arrow Flight Catalog implementation: #59270. The merged Doris Catalog path is #55274/#57898.
- Logging experiment: #61990.
- DDL regression fix: #62018.
- 4.0 result-receiver backport: #63154.
- Array child nullability and focused IPC tests: #64976 and #65044.
- LargeUtf8 offset fix attempt: #62681.
- Early naive-DATETIME attempt: #65823, superseded by #66873.

## Maintenance rules

- Add each new issue or PR under the closest topic and link it back to this tracker.
- Keep the primary fix and all required release-branch backports on the same line.
- Check an item only after the fix is merged and the affected supported versions are verified.
- If a stale-closed report still reproduces, reopen it or create a replacement and link both directions.
- Record user-visible protocol limitations in the Arrow Flight SQL documentation, not only in regression-test skips.

Start by selecting one unchecked item and reading its linked issue or referenced pull request, such as #67364, #67368, or #62538; this tracker does not name a source file or test entry point. Done means reproducing the selected problem, defining or linking the owning work, and verifying the fix or documenting the remaining limitation.

Written by the indexing model from the issue text.