Add execution diagnostics for failed SPL scripts
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 4.7k
- Forks
- 364
- PR merge metrics
- No merged PRs in 30d
Description
Problem
When an SPL script fails during execution, identifying the actual source of the problem can require manually inspecting the script and reproducing the failure.
For data-processing workflows, this can become difficult when a script contains multiple dependent operations/cells. A generic exception message or stack trace may not provide enough execution context to quickly answer:
- Which SPL cell/operation failed?
- How long had the pipeline been running before the failure?
- What was the input/output size of the failing operation?
- Which previous operation produced the input?
- What data condition may have contributed to the failure?
This can increase the time required to diagnose failures in larger SPL data-processing workflows.
Proposed Feature
I would like to explore adding an SPL Execution Inspector that records lightweight execution information for SPL operations/cells.
For example:
SPL Execution
A1 READ ✓ 1.2s 1,250,432 rows
A2 FILTER ✓ 0.4s 843,219 rows
A3 GROUP ✓ 2.8s 51,203 rows
A4 JOIN ✗ 0.1s -
Failure:
Cell: A4
Operation: JOIN
Input: A3
Status: FAILED
Exception: ...
The initial version could focus on:
- Identifying the failing cell/operation.
- Recording execution status.
- Recording execution duration.
- Capturing structured exception information.
- Showing the execution path leading to the failure.
A later extension could provide data-quality diagnostics, such as identifying null or unexpected values in relevant input fields.
Expected User / Business Impact
The goal is not only to provide a nicer error message, but to reduce the time required to diagnose failed data-processing workflows.
For example, instead of:
Error: NullPointerException
the user could see:
A17 JOIN failed
Input:
A15 → 1,250,432 rows
A16 → 43,821 rows
Potential issue:
customerId contains null values in A15
Next diagnostic:
Inspect rows where customerId is null
This could help developers and data engineers identify the problematic transformation and investigate the underlying data issue without manually tracing the entire script.
Scope
For an initial contribution, I would keep the scope limited to execution tracing and failure context rather than attempting to build a complete debugger.
I would first investigate the current execution architecture, identify the appropriate instrumentation points, and implement a small prototype before expanding the feature.
Questions for Maintainers
Before starting implementation, I would appreciate feedback on:
- Is execution tracing/failure diagnostics a useful direction for esProc?
- Is there an existing mechanism for obtaining cell-level execution information that I should extend instead?
- Which part of the execution engine/IDE would be the appropriate integration point?
- Would a diagnostic feature like this fit the project's current architecture and roadmap?
I would be happy to investigate the codebase and submit an initial implementation based on the maintainers' feedback.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by investigating the current execution architecture and any existing cell-level execution information, then identify instrumentation points in the execution engine or IDE. A first prototype would be done when it reports the failing cell or operation, status, duration, structured exception information, and the execution path leading to failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100