apache / apache/hudi

[Milestone 3] Dynamic inline/out-of-line writer support

Open
#18,112 1 comment 0 reactions 2 assignees Claimed by @rahil-c View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**
The writer path needs to be updated to include the ability to write out the incoming, inline blob data ~~to a container file that will hold one or more of these byte arrays~~. (container files are no longer within the scope of blobs we will not be pursuing container files.)

~~The decision of how to store the data (inline vs out-of-line) will be determined by the configurations described in the RFC. When a file is created by the Hudi writer, it is automatically considered a managed file.~~

Out-of-line: Users provide a file path in the blob struct pointing to the actual blob.
in-line: Users write the blob bytes directly during ingestion.

**Why this task is needed:**
Allows the user to easily offload large byte arrays to files to reduce the storage size for the structured columns in their dataset.

### Task Type

Code improvement/refactoring

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.