[Bug] Cross-cluster Catalog with Arrow Flight fails to read Variant type
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
**Environment**
- **Doris version:** 4.0.3
- **Deployment:** Kubernetes (k8s) Operator
- **Cluster Type:** `DorisDisaggregatedCluster` (Cloud Mode / Storage-Compute separation)
- **Configuration:** Catalog property `"use_arrow_flight" = "true"` is set.
### What's Wrong?
**Describe the bug**
In Doris 4.0.3, when using a cross-cluster Catalog (`type = 'doris'`) with the Arrow Flight protocol enabled (`use_arrow_flight = 'true'`) to read data from a remote Doris cluster, querying a table that contains a `Variant` column fails. The query throws a `NOT_IMPLEMENTED_ERROR`. It appears the underlying Arrow reader currently does not support the serialization/deserialization of the `Variant` data type.
**To Reproduce**
1. Create a Catalog connecting to a remote Doris cluster and force enable Arrow Flight:
```sql
CREATE CATALOG remote_doris PROPERTIES (
"type" = "doris",
"use_arrow_flight" = "true",
"fe_http_hosts" = "...",
"fe_arrow_hosts" = "...",
"fe_thrift_hosts" = "...",
"user" = "...",
"password" = "..."
);
```
2. Ensure the target table being queried on the remote cluster contains a column of type `Variant` (e.g., `ext_fields`).
3. Execute a query from the local cluster via this Catalog:
```sql
SELECT ext_fields FROM remote_doris.db.table LIMIT 10;
```
4. The following error is triggered:
```text
ERROR 1105 (HY000): errCode = 2, detailMessage = (...)[NOT_IMPLEMENTED_ERROR]read_column_from_arrow with type variant. cur path: /dummyPath
```
*(Note: Even if attempting to use `CAST(ext_fields AS STRING)` in the query, the conversion cannot be pushed down before the error occurs, triggering the same interception error.)*
### What You Expected?
The Arrow Flight transmission link should natively support the cross-cluster reading and parsing of `Variant` types, allowing the query to return data successfully without throwing a `NOT_IMPLEMENTED_ERROR`.
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.