apache / apache/hop

[Feature Request]: Consider a binary file data type

Open
#2,341 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage Metadata P3
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
19h
Merged PRs (30d)
227

Description

### What would you like to happen?

Migration: https://issues.apache.org/jira/browse/HOP-4239

We could consider a new value type called "Binary file" which stores large data sets in (temporary) files instead of keeping data in memory.

This would solve use-cases where large amounts of data need to be transferred from one location to another.  

For example, Oracle BLOB fields have the ability to contain up to 4GB or data.  That is more than fits in a Binary field and it would be a shame to waste memory on it in the first place.

When writing the data to another database or location you can stream the data back out from the file.

The trick would be that we would need a way to clean the temporary files back up at the end of a pipeline.  The constructor of the ValueMetaBinaryFile() would need to have a hook to a mechanism to remove the file when no longer needed.

### Issue Priority

Priority: 3

### Issue Component

Component: Metadata

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the HOP-4239 migration and the existing ValueMetaBinaryFile() constructor referenced in the issue. Trace how metadata values move through a pipeline and how temporary resources are cleaned up; done means a design and implementation that supports large binary data, streaming, and reliable cleanup at pipeline end.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.