[Feature Request]: Consider a binary file data type
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 19h
- Merged PRs (30d)
- 227
Description
### What would you like to happen?
Migration: https://issues.apache.org/jira/browse/HOP-4239
We could consider a new value type called "Binary file" which stores large data sets in (temporary) files instead of keeping data in memory.
This would solve use-cases where large amounts of data need to be transferred from one location to another.
For example, Oracle BLOB fields have the ability to contain up to 4GB or data. That is more than fits in a Binary field and it would be a shame to waste memory on it in the first place.
When writing the data to another database or location you can stream the data back out from the file.
The trick would be that we would need a way to clean the temporary files back up at the end of a pipeline. The constructor of the ValueMetaBinaryFile() would need to have a hook to a mechanism to remove the file when no longer needed.
### Issue Priority
Priority: 3
### Issue Component
Component: Metadata
Contributor guide
Research direction
Start by reviewing the HOP-4239 migration and the existing ValueMetaBinaryFile() constructor referenced in the issue. Trace how metadata values move through a pipeline and how temporary resources are cleaned up; done means a design and implementation that supports large binary data, streaming, and reliable cleanup at pipeline end.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100