[Milestone 3] Dynamic inline/out-of-line writer support
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Task Description
**What needs to be done:**
The writer path needs to be updated to include the ability to write out the incoming, inline blob data ~~to a container file that will hold one or more of these byte arrays~~. (container files are no longer within the scope of blobs we will not be pursuing container files.)
~~The decision of how to store the data (inline vs out-of-line) will be determined by the configurations described in the RFC. When a file is created by the Hudi writer, it is automatically considered a managed file.~~
Out-of-line: Users provide a file path in the blob struct pointing to the actual blob.
in-line: Users write the blob bytes directly during ingestion.
**Why this task is needed:**
Allows the user to easily offload large byte arrays to files to reduce the storage size for the structured columns in their dataset.
### Task Type
Code improvement/refactoring
### Related Issues
**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.