apache / apache/hop

[Feature Request]: Create an enriched error capture metadata for pipelines and workflows

Open
#7,010 1 comment 0 reactions 0 assignees View on GitHub
awaiting triage Metadata P3 Transforms
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 32m
Merged PRs (30d)
216

Description

### What would you like to happen?

Description:
Apache Hop already provides logging and error handling features at different levels, but it still lacks a centralized and standardized mechanism to capture failures with full execution context.

The proposal is to add a new metadata type, or an equivalent feature, that allows defining an error trigger for pipelines and workflows. Whenever a failure occurs, this trigger should invoke a dedicated error-handling pipeline and send a structured payload containing as much execution context as possible.

Objective:
Enable full observability and analysis of failures, with enough information for auditing, troubleshooting, monitoring, and possible reprocessing.

Expected error event payload:

project and environment;
source workflow or pipeline name;
transform or action that failed;
error message;
stack trace or technical cause, when available;
timestamp;
error status and level;
relevant parameters and variables;
log channel / execution id;
error row metadata, when applicable;
row data that caused the error, when available.

Expected behavior:

configure a reusable error pipeline through metadata;
allow usage at both pipeline and workflow levels;
support the distinction between row-level errors, pipeline execution errors, and workflow errors;
send partial payloads when some details are not available;
allow future extension for sensitive data masking, payload limits, and retry policies.

Benefits:

centralized error handling;
standardized observability;
faster diagnosis;
better operational governance;
foundation for alerts, auditing, and reprocessing.

Note:
The goal is not to replace the current logging mechanisms, but to complement them with a dedicated enriched failure capture feature.

### Issue Priority

Priority: 3

### Issue Component

Component: Transforms, Component: Metadata

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing Apache Hop's existing logging, error-handling, pipeline, workflow, and metadata mechanisms, then define the scope and design for reusable error triggers and structured partial payloads across row, pipeline, and workflow failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.