[Feature Request]: Create an enriched error capture metadata for pipelines and workflows
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
Description:
Apache Hop already provides logging and error handling features at different levels, but it still lacks a centralized and standardized mechanism to capture failures with full execution context.
The proposal is to add a new metadata type, or an equivalent feature, that allows defining an error trigger for pipelines and workflows. Whenever a failure occurs, this trigger should invoke a dedicated error-handling pipeline and send a structured payload containing as much execution context as possible.
Objective:
Enable full observability and analysis of failures, with enough information for auditing, troubleshooting, monitoring, and possible reprocessing.
Expected error event payload:
project and environment;
source workflow or pipeline name;
transform or action that failed;
error message;
stack trace or technical cause, when available;
timestamp;
error status and level;
relevant parameters and variables;
log channel / execution id;
error row metadata, when applicable;
row data that caused the error, when available.
Expected behavior:
configure a reusable error pipeline through metadata;
allow usage at both pipeline and workflow levels;
support the distinction between row-level errors, pipeline execution errors, and workflow errors;
send partial payloads when some details are not available;
allow future extension for sensitive data masking, payload limits, and retry policies.
Benefits:
centralized error handling;
standardized observability;
faster diagnosis;
better operational governance;
foundation for alerts, auditing, and reprocessing.
Note:
The goal is not to replace the current logging mechanisms, but to complement them with a dedicated enriched failure capture feature.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms, Component: Metadata
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing Apache Hop's existing logging, error-handling, pipeline, workflow, and metadata mechanisms, then define the scope and design for reusable error triggers and structured partial payloads across row, pipeline, and workflow failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100