apache / apache/paimon

[Feature] Support reading actual data from blob-descriptor-field in spark

Open
#9,582 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

blob-descriptor-field stores blob values as serialized BlobDescriptor bytes inline in data files. In Spark, reading such fields as normal binary data should return the actual blob bytes, while blob-as-descriptor=true should return descriptor bytes.

Currently, spark may fail to resolve the actual blob data because the read path does not always attach the required descriptor reader to the reconstructed blob value.

### Solution

Pass a table-level UriReaderFactory through the spark scan/read path, and wrap read rows with a descriptor-resolving row when blob-as-descriptor=false.

The wrapper reattaches the proper reader to BlobRef values, so spark can read actual blob bytes from blob-descriptor-field. When blob-as-descriptor=true, spark keeps returning serialized descriptor bytes.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Spark scan/read path and where a table-level UriReaderFactory can be carried through it. Compare reads with blob-as-descriptor=false and true, using the blob-descriptor-field behavior described in the issue. Done means normal binary reads return actual blob bytes while descriptor mode still returns serialized descriptor bytes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.