SPLWare / SPLWare/esProc

Add execution diagnostics for failed SPL scripts

Open
#68 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
4.7k
Forks
364
PR merge metrics
No merged PRs in 30d

Description

Problem

When an SPL script fails during execution, identifying the actual source of the problem can require manually inspecting the script and reproducing the failure.

For data-processing workflows, this can become difficult when a script contains multiple dependent operations/cells. A generic exception message or stack trace may not provide enough execution context to quickly answer:

  • Which SPL cell/operation failed?
  • How long had the pipeline been running before the failure?
  • What was the input/output size of the failing operation?
  • Which previous operation produced the input?
  • What data condition may have contributed to the failure?

This can increase the time required to diagnose failures in larger SPL data-processing workflows.

Proposed Feature

I would like to explore adding an SPL Execution Inspector that records lightweight execution information for SPL operations/cells.

For example:

SPL Execution

A1   READ       ✓   1.2s    1,250,432 rows
A2   FILTER     ✓   0.4s      843,219 rows
A3   GROUP      ✓   2.8s       51,203 rows
A4   JOIN       ✗   0.1s             -

Failure:
  Cell: A4
  Operation: JOIN
  Input: A3
  Status: FAILED
  Exception: ...

The initial version could focus on:

  1. Identifying the failing cell/operation.
  2. Recording execution status.
  3. Recording execution duration.
  4. Capturing structured exception information.
  5. Showing the execution path leading to the failure.

A later extension could provide data-quality diagnostics, such as identifying null or unexpected values in relevant input fields.

Expected User / Business Impact

The goal is not only to provide a nicer error message, but to reduce the time required to diagnose failed data-processing workflows.

For example, instead of:

Error: NullPointerException

the user could see:

A17 JOIN failed

Input:
  A15 → 1,250,432 rows
  A16 → 43,821 rows

Potential issue:
  customerId contains null values in A15

Next diagnostic:
  Inspect rows where customerId is null

This could help developers and data engineers identify the problematic transformation and investigate the underlying data issue without manually tracing the entire script.

Scope

For an initial contribution, I would keep the scope limited to execution tracing and failure context rather than attempting to build a complete debugger.

I would first investigate the current execution architecture, identify the appropriate instrumentation points, and implement a small prototype before expanding the feature.

Questions for Maintainers

Before starting implementation, I would appreciate feedback on:

  1. Is execution tracing/failure diagnostics a useful direction for esProc?
  2. Is there an existing mechanism for obtaining cell-level execution information that I should extend instead?
  3. Which part of the execution engine/IDE would be the appropriate integration point?
  4. Would a diagnostic feature like this fit the project's current architecture and roadmap?

I would be happy to investigate the codebase and submit an initial implementation based on the maintainers' feedback.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by investigating the current execution architecture and any existing cell-level execution information, then identify instrumentation points in the execution engine or IDE. A first prototype would be done when it reports the failing cell or operation, status, duration, structured exception information, and the execution path leading to failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.