apache / apache/doris

[Bug] Arrow Flight SQL: Corrupted StringArray serialization results in empty DataBuffer and invalid Offsets

Closed
#61,124 1 comment 0 reactions 0 assignees View on GitHub
Stale
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

doris: 4.0.3
adbc client: github.com/apache/arrow-go/v18

### What's Wrong?

#### Description:
When fetching large JSON string columns using Arrow Flight SQL, the returned RecordBatch contains corrupted memory state. Specifically, the Offsets indicate valid data ranges, but the DataBuffer is reported as size 0.

#### Evidence:
- When accessing the last row of a batch:
- Total Length: 1283
- Offsets Buffer Length: 1284
- Last 10 Offsets: [... 1251873, 0] (Note the non-monotonic reset to 0)
- Data Buffer Total Size: 0 bytes

This indicates that during the serialization of the RecordBatch in the Doris BE, the DataBuffer was either truncated or released prematurely, while the Offsets metadata was not correctly synchronized.

#### Impact:
Client-side library (like apache/arrow-go) encounters a slice bounds out of range panic when attempting to access the string value, as it tries to slice a 0-length buffer using offsets that point to invalid memory.

### What You Expected?

Investigate the ArrowFlightStream serialization logic in the BE, particularly how StringArray offsets are calculated and how the DataBuffer lifecycle is managed during stream fragmentation.

### How to Reproduce?

1. Execute a query via Arrow Flight SQL that returns a large number of rows (e.g., > 2000 rows).
2. The target column contains JSON string data (average length ~1KB per row).
3. Use reader.Next() to iterate through RecordBatches.
4. Access the last few rows of a specific RecordBatch using array.String.Value(i).

### Anything Else?

--- [DEBUG] Arrow Array Memory State Dump ---
Array Type: *array.String
Total Length: 1283
Null Count: 0
Offsets Buffer Length: 1284
Last 10 Offsets: [1244061 0 1246014 0 1247967 0 1249920 0 1251873 0]
Data Buffer Total Size (Bytes): 0
Target Row: 1282 | StartOffset: 1251873, EndOffset: 0

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the ArrowFlightStream serialization logic in the Doris BE, then reproduce the issue with a large JSON string query and inspect batches through reader.Next() and array.String.Value(i). Compare the offsets and data buffer for the final rows; done means the returned RecordBatch no longer produces inconsistent offsets or a slice-bounds panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
api, backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.