[Feature] Support reading actual data from blob-descriptor-field in spark
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
blob-descriptor-field stores blob values as serialized BlobDescriptor bytes inline in data files. In Spark, reading such fields as normal binary data should return the actual blob bytes, while blob-as-descriptor=true should return descriptor bytes.
Currently, spark may fail to resolve the actual blob data because the read path does not always attach the required descriptor reader to the reconstructed blob value.
### Solution
Pass a table-level UriReaderFactory through the spark scan/read path, and wrap read rows with a descriptor-resolving row when blob-as-descriptor=false.
The wrapper reattaches the proper reader to BlobRef values, so spark can read actual blob bytes from blob-descriptor-field. When blob-as-descriptor=true, spark keeps returning serialized descriptor bytes.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Spark scan/read path and where a table-level UriReaderFactory can be carried through it. Compare reads with blob-as-descriptor=false and true, using the blob-descriptor-field behavior described in the issue. Done means normal binary reads return actual blob bytes while descriptor mode still returns serialized descriptor bytes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100