[Tracking] Protocol-agnostic session and execution layer: MySQL and Arrow Flight SQL as equal front ends

This issue tracks the work that makes MySQL and Arrow Flight SQL **equal front ends over one session
layer** in the FE: one session object, one result-encoding abstraction, one Doris-to-Arrow type
mapping, one connection pool and quota, and one prepared-statement registry.

Part of #65615. The design is being written up as a DSIP; this issue is the implementation tracker.
If you hit one of the problems listed here, please open a separate bug report and link it back.

#67578 builds on this one: completing the Arrow Flight SQL server and shipping an official Doris
ADBC driver needs the session lifecycle, the result path and the prepared-statement registry that
this issue unifies. The two can be discussed in parallel, but that work only lands cleanly on top
of what is tracked here.

**Relationship to the earlier protocol SPI work.** #60355 / #60361 (both closed) decoupled the
*listeners*: how a protocol server is discovered, configured and started. This issue covers
everything behind the listener - session state, the result path, type mapping, connection governance
and prepared statements - which is still written against the MySQL protocol and special-cased for
Arrow Flight SQL.

## Why

An Arrow Flight SQL session already *is* a `ConnectContext`, `SHOW PROCESSLIST` lists it and `KILL`
works on it. What is not shared is most of what the execution layer does afterwards. A read of the
FE session/connection/result paths and the BE result sinks turned up the following; items with a PR
have been verified locally, the rest are code-reading conclusions and are marked as such.

- The execution layer branches on the protocol in roughly 40 places (`StmtExecutor` 18,
`ConnectProcessor` 8, nine `Command`s, plus `Coordinator`, `NereidsCoordinator`,
`StatementContext`, `FrontendServiceImpl`) and reaches for `MysqlChannel` in roughly 50. A Flight
session throws from `getMysqlChannel()`, so every unguarded reach is a live failure rather than a
style problem - see #67569 for one that reached production.
- There are **three** Doris-to-Arrow type mappings: the BE's `convert_to_arrow_type`, the FE's
`FlightSqlSchemaHelper.getArrowType` (used by `CommandGetTables`), and `FlightSqlChannel`, which
encodes every FE-side result as `utf8`. The FE mirror already disagrees with the BE in four types.
- Flight connections are governed by their own pool with their own limit
(`arrow_flight_max_connections`) plus a per-user token LRU, so `qe_max_connection` and
`max_user_connections` do not apply to them, and a live session that the LRU evicts surfaces to the
client as `invalid bearer token`.
- Flight prepared statements do not use the FE's prepared-statement machinery:
`createPreparedStatement` stores the SQL text and returns a placeholder schema, and parameter
binding is not implemented.

## Status convention

Same as #65615: `[x]` means merged or confirmed complete; `[ ]` means open or needs follow-up.

## Stage 0 - independent bug fixes

Defects found by the survey that do not depend on the refactor. All three are merged and carry
`dev/4.2.x`, so they reach the 4.2 branch ahead of the refactor.

- [x] **A Flight session cannot forward a statement to the master FE:** #67569, merged 2026-09-09.
`FEOpExecutor.buildStmtForwardParams()` reads `CLIENT_DEPRECATE_EOF` straight off the MySQL channel
(introduced by #61050), so on a multi-FE deployment every statement a Flight connection has to
forward - any DDL issued to a follower or observer, or anything at all under
`force_forward_all_queries` - fails with `getMysqlChannel not in mysql connection`.
- [x] **`ConnectType` declared twice, plus an unreachable dispatch on the forward path:** #67572,
merged 2026-09-08. Cleanup, no behavior change. One leftover, deliberately deferred to the
`ProtocolAdapter` PR: `ConnectProcessor.connectType` is still a redundant mirror of
`ctx.getConnectType()`.
- [x] **A Flight session reports `0.0.0.0:0` as its client address:** #67576, merged 2026-09-10.
The `Host` column of
`SHOW PROCESSLIST` and `information_schema.processlist`, the audit log's `client_ip`, and the kill
and timeout warnings all showed the placeholder, although the real address is resolved when the
bearer token is issued.

## Stage 1 - one session, one result path

Behavior-preserving refactor, guarded by golden tests that pin the MySQL packet bytes.

- [x] Golden tests first: #67789, merged 2026-09-11. A recording `MysqlChannel` drives a statement set through
`MysqlConnectProcessor`/`StmtExecutor` and the produced packets are compared byte for byte; the same
statements then run through the Flight processor and their schema and rows are compared. This is the
safety net for everything below, so it lands before any refactor. 27 MySQL cases and 8 Flight
statements, recorded as annotated hexdumps rather than binary blobs so a diff is reviewable. Two
responses are recorded as shape rather than bytes, because their payload moves with unrelated
changes: a parser error carries the grammar's whole keyword list, and an `EXPLAIN` carries the
current plan text. `COM_STMT_EXECUTE` is out of scope for the golden files because
`supportHandleByFe()` is false for it, so its result always comes from a BE - the cursor-fetch
packet boundaries stay covered by `prepared_stmt_p0` instead.
- [x] `ProtocolAdapter`: #67835, merged 2026-09-11. The per-protocol half of `ConnectContext`
(channel, capabilities, accept loop, close, the `COM_STMT_EXECUTE` state, the Flight result cache,
endpoints and deferred executors) moves behind an interface with a MySQL and a Flight
implementation; `ConnectContext.forMysql` / `forMysqlProxy` / `forFlight` create the bound
context. `ConnectContext` keeps its class name and getter signatures - it is referenced by ~900
files - and `FlightSqlConnectContext` goes away. Flight commands get a per-session lock:
`ConnectContext` is not thread safe and Flight did not serialize a session's calls. Verified
against the #67789 baseline: not a byte changed. The internal (no-client) context stays a MySQL
context over a `DummyMysqlChannel` for now, because the execution layer still keys its result
path on `ConnectType`; a distinct internal adapter follows once those branches are gone.
- [x] Package move: #67866, merged 2026-09-11. `org.apache.doris.service.arrowflight` becomes
`org.apache.doris.arrowflight`. Flight was filed under `service` in 2023 (#24772) next to the thrift
`FrontendServiceImpl`; it is now a peer of the top-level `mysql` package, and #67835 already had to
mirror a `protocol` sub-package on both sides. A pure `git mv` (36 files, only `package`/`import`
lines change, nothing references the old name by string), sequenced before `ResultSender` so the
new implementations land in the final place.
- [x] `ResultSender`: #67883, merged 2026-09-12. The result-encoding half moves out of
`StmtExecutor`/`ConnectProcessor`: `qe.protocol.ResultSender` with `MysqlResultSender` /
`FlightResultSender`, `StmtExecutor` without a serializer field or `MysqlChannel` parameters, the
three text-result paths (`EXPLAIN`, `EXPLAIN PLAN PROCESS`, `REPLAY`) through the one
`sendResultSet`, and `ConnectProcessor` without its `connectType` field (the per-statement
protocol work of a multi-statement request is `adapter.finishStatement`). The MySQL golden is
byte-identical; the Flight golden changes in exactly one entry, `EXPLAIN PLAN PROCESS`, which
had no Flight branch before and now returns its rows. Also carries the leftovers of the #67835
review (lock wait bound = `getExecTimeoutS()`, `FlightRuntimeException` passed through the
producer's catch-alls, a WARN when the lock is given up).
- [ ] Remaining protocol branches in the execution layer go to zero: #67900, opened 2026-09-13.
`StmtExecutor`, `ConnectContext.supportHandleByFe`, the nine insert / transaction `Command`s, both
coordinators, the short-circuit rule and `FEOpExecutor` stop asking `getConnectType()`;
`ProtocolAdapter` gains one method per use: `canReplayForwardedQueryResult`, `supportsFeSideResult`,
`supportsShortCircuitPointQuery`, `canRetryQuery`, the lifecycle hooks `beforeStatement` /
`beforeQuery` / `returnsResultFromLocal`, and `fillForwardRequest` (with
`MysqlProtocolAdapter.restoreFromForwardRequest` as the master's side). The Flight-only
`returnResultFromLocal` flag is no longer flipped from outside the adapter, and `FlightResultSender`
no longer has to undo a flip for `EXPLAIN`. One deliberate byte change, recorded in the MySQL
golden: the channel is reset once, when a statement starts, instead of in the query path and in
nine commands, so a request from a client without `CLIENT_MULTI_STATEMENTS` whose last statement
is not a query (`select 1; set @a = 1`) delivers only the last response instead of the buffered
result set of the `SELECT` followed by an `OK` -- a stream no client parses. The recording channel
of the golden now models the send buffer (a reset drops what was written after the last flush), so
the golden shows what reaches the client. The Flight golden is byte-identical.

#67520 (merged 2026-09-10) lands in the middle of this surface: it touches `ConnectProcessor`,
`MysqlConnectProcessor`, `StmtExecutor`, `FEOpExecutor`, `MysqlProto` and `ConnectContext`, and adds
`MysqlCursorFetchCompatibility`, a client-capability branch that decides where a cursor result's
terminators go. The adapter extraction absorbs it rather than works around it: that logic belongs to
the MySQL adapter (#67835 moved it there as `MysqlProtocolAdapter.clientConsumesCursorMetadataTerminator`).
#67753 (merged 2026-09-11, right after #67835) adds one more thing the `ResultSender` step has to
carry: an internal query run on behalf of a client (the IVM dry run) streams its rows to that
client's MySQL channel, so `StmtExecutor` grew channel-taking overloads of its send methods. The
result sender is handed over by the caller in that case rather than taken from the executor's own
session.

## Stage 2 - one type mapping

- [ ] One enumerable Doris-to-Arrow mapping, shared by a table-driven BE unit test and an FE test that
validates the FE mirror against the same golden file, and the four known FE/BE disagreements fixed.
#66761 added TIMESTAMP_NS to both mappings separately - one more line in `arrow_row_batch.cpp` and
one in `FlightSqlSchemaHelper` - which is exactly the drift this stage removes; the golden file
covers it.
- [ ] Field metadata completed on the BE: `doris_type` for every non-native type plus
`ARROW:extension:name` (`doris.largeint`, `doris.ipv4`, `doris.ipv6`, `doris.bitmap`, `doris.hll`,
`doris.quantile_state`, `doris.agg_state`, `doris.time`), with JSONB and VARIANT using the canonical
`arrow.json` extension. **Storage types do not change**: `convert_to_arrow_type` also serves the
Spark and Flink connectors' read path and Python UDFs, so changing them would change those
protocols. In particular LARGEINT stays a decimal string rather than becoming `decimal128(38,0)`,
which cannot represent an int128 (39 decimal digits; Arrow caps decimal128 precision at 38), and
IPV4/IPV6 stay `int32`/`utf8` rather than `fixed_size_binary`.
- [ ] FE-side results (`SHOW`, `EXPLAIN`, replayed proxy results) built as typed vectors instead of
all-`utf8`.
- [ ] A cross-protocol regression suite: the same query over JDBC and over Flight, compared column by
column across the whole type matrix.

## Stage 3 - one connection pool and quota

Builds on #67504 (merged 2026-09-07), which stopped a finished Flight query from holding its
coordinator until `wait_timeout`.

- [ ] Single pool: `FlightSqlConnectPoolMgr` folds into `ConnectPoolMgr`, and Flight registration goes
through `qe_max_connection` and `max_user_connections` like any other connection.
- [ ] The bearer token becomes a session credential: no token cache, no per-user LRU, no separate
lifetime. `arrow_flight_token_cache_size` and `arrow_flight_token_alive_time_second` are deprecated
with a warning for one major version, and `arrow_flight_max_connections` becomes a sub-quota that
defaults to following `qe_max_connection`.
- [ ] A `Protocol` column on `SHOW PROCESSLIST` and `information_schema.processlist` (FE and the BE
schema scanner), plus a `protocol` field in the audit log.

## Stage 4 - prepared statements

- [ ] The prepared-statement registry is extracted from the MySQL `COM_STMT_*` handling so both front
ends share it.
- [ ] Flight `createPreparedStatement`/`getFlightInfoPreparedStatement`/`closePreparedStatement` go
through it and return the real result and parameter schemas.
- [ ] `acceptPutPreparedStatementQuery` binds Arrow parameter batches, so Flight SQL JDBC with
`useServerPrepStmts=true` and the ADBC default path work.

## Known gaps, not yet scheduled

Code-reading conclusions from the same survey. They are not blocking the stages above; each needs its
own report or PR.

- [ ] **A forwarded statement's result set is dropped over Flight.** Even after #67569, a statement
that is forwarded to the master and returns rows gives the Flight client nothing, because the proxy
result is replayed by `ConnectProcessor.finalizeCommand()`, which is MySQL-only. Affects only
statements that both need forwarding and carry a result set.
- [ ] **`arrow_flight_token_alive_time_second` is applied in the wrong unit.** `createToken()` converts
the configured seconds as if they were minutes, so the recorded expiry is 60x too far out; the token
is actually evicted by the Guava cache's own expiry, which makes the explicit expiry check dead code.
- [ ] **`AGG_STATE` is advertised as `utf8`** while carrying non-UTF-8 bytes - the same shape as the
Iceberg `BINARY` problem in #67371.
- [ ] **`select 1` and `select @@var` always go to the BE on a Flight session**, because
`supportHandleByFe()` is hard-coded false for Arrow Flight SQL. Enabling FE-side results for Flight
is deliberately out of scope until the FE result path is typed (Stage 2).

## Upstream PRs this work depends on

- [x] #67504 - release a finished Flight query's coordinator instead of holding it until
`wait_timeout`. Merged 2026-09-07; unblocks Stage 1.
- [ ] #67530 - keep the Doris type of a nested LARGEINT in the Arrow schema. Under a requested BE
type-layer refactor; Stage 2 waits on the outcome so the two do not collide. No word on the scope
or the timing of that refactor since 2026-09-05, so Stage 2 stays parked while Stage 1 proceeds.
- [ ] #65789 - utf8/large-string schema-array mismatch for oversized columns. Stage 2 rebases on it
rather than touching `arrow_block_convertor.cpp`.

## Progress

- 2026-09-07: survey and design complete; Stage 0 PRs submitted.
- 2026-09-08 .. 2026-09-10: Stage 0 merged in full (#67572, #67569, #67576), each labelled
`dev/4.2.x`.
- 2026-09-10: Stage 1 opened with the golden baseline, #67789.
- 2026-09-11: #67789 merged. The recorded packets are now the gate for the rest of Stage 1: each
refactor PR runs both golden tests before and after, and since those PRs are behavior-preserving,
a changed byte means the refactor changed something it should not have. Stage 2 remains parked
on #67530.
- 2026-09-11: #67835 opened and merged the same day - the `ProtocolAdapter` extraction, first
refactor PR of Stage 1. Both golden tests pass unchanged on it. Next is the `ResultSender`
extraction, which also has to absorb the channel-taking send overloads #67753 added.
- 2026-09-11: #67866 opened and merged the same day - the `service.arrowflight` -> `arrowflight`
package move, a mechanical PR slotted in before `ResultSender`. Flight now lives in the top-level
`org.apache.doris.arrowflight` package, a peer of `org.apache.doris.mysql`; `ResultSender` is next
and its implementations go to `mysql.protocol` / `arrowflight.protocol`.
- 2026-09-12: #67883 opened - the `ResultSender` extraction, second refactor PR of Stage 1. Only
the expected Flight golden entry changed (`EXPLAIN PLAN PROCESS` gains a result); everything
MySQL is byte for byte. What remains for Stage 1 is the capability-predicate PR (the branches
outside the result path) and the performance baseline.
- 2026-09-12: #67883 merged. `StmtExecutor` and `ConnectProcessor` no longer encode anything
themselves; every protocol branch left in the execution layer is now a candidate for a capability
predicate on `ProtocolAdapter`, and that PR is next. The list, on the merged master: five
`ConnectType` checks in `StmtExecutor` (including the Flight forward refusal that #67569 added),
`supportHandleByFe()`, the nine `Command`s that reset the MySQL channel, the two coordinators and
`QueryProcessor`, `StatementContext.close`, the short-circuit rule and `FEOpExecutor`. The
performance baseline is still pending.
- 2026-09-13: #67900 opened - the capability-predicate PR, last refactor PR of Stage 1. After it,
`grep 'ConnectType\.\|getMysqlChannel()'` over `qe/**` and `nereids/**` (outside `*/protocol/`)
finds only the `ConnectContext.getMysqlChannel()` delegate and two lines of `MysqlConnectProcessor`
that read the client's packets. Two things found on the way, filed for their own fixes rather than
folded in: on a Flight session, `SHOW TABLES / DATABASES / COLUMNS / TABLE STATUS ... WHERE` run an
internal query on the session's own context, so the plan gets an Arrow result sink while the
frontend tries to pull the rows over `fetch_data` (the request hangs until it times out); and
`MysqlChannel.reset()` does not rewind the sequence id, so the response a client without
`CLIENT_MULTI_STATEMENTS` gets for `select 1; select 2` starts at sequence id 4, which pymysql and
libmysqlclient reject (Connector/J does not check). The internal adapter for the no-client context
and the performance baseline remain.

Start with the open Stage 1 work in StmtExecutor, ConnectContext, ProtocolAdapter, the coordinators, and FEOpExecutor, using the #67789 golden tests as the baseline. Then review the Stage 2 and Stage 3 requirements, including arrow_row_batch.cpp, FlightSqlSchemaHelper, and FlightSqlConnectPoolMgr. Done means the listed protocol branches, type mappings, connection governance, and prepared-statement paths are unified without breaking the MySQL and Flight regression comparisons.

Written by the indexing model from the issue text.