apache / apache/hop

[Feature Request]: add error handling to Metadata Injection

Open
#2,419 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage P3 Transforms
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 32m
Merged PRs (30d)
216

Description

### What would you like to happen?

Migration: https://issues.apache.org/jira/browse/HOP-4503

In my use case, I have a pipeline that is being injected that ends with a db table load (specifically PostgreSQL Bulk loader).  When this process fails due to a timeout, there isn't a good way to capture the affected table that did not load. Passing some metadata from the injected pipeline would be beneficial in capturing that information to attempt a re-process.

### Issue Priority

Priority: 3

### Issue Component

Component: Transforms

Contributor guide

Open the contributing guide

Research direction

Start with migration HOP-4503 and trace the Metadata Injection flow through the injected pipeline and PostgreSQL Bulk loader scenario. Determine how a timeout failure could expose the affected table and pass metadata back to the caller. Done means the failed table can be identified and its metadata used to support re-processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.