apache / apache/hop

[Feature Request]: Dev Cache Mode / Step Checkpoints (development only)

Open
#6,537 2 comments 1 reaction 0 assignees View on GitHub
awaiting triage Hop Gui P3
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 32m
Merged PRs (30d)
216

Description

### What would you like to happen?

Goal
Speed up development and debugging cycles in Apache Hop pipelines by reusing a step output as a “checkpoint”, avoiding reprocessing the entire upstream on every run.

Concept (proposal)

The user marks a step as a checkpoint.

Hop stores the step output (rows/dataset) in a local cache.

On later runs, Hop can skip upstream steps and start from the checkpoint, loading data from cache.

The user explicitly chooses:

Run from the beginning

Run from the last checkpoint

Note
Unlike a “test pipeline”, the idea is not to create dataset metadata. This is a development-only cache feature to speed up iteration, especially when upstream extraction (e.g., from a database) is slow. The user can define where to store the cache and how many rows to save. At runtime, they can run from the beginning or resume from the last checkpoint, without creating extra metadata or separate test pipelines.

### Issue Priority

Priority: 3

### Issue Component

Component: Hop Gui

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named in the issue. Start by locating the Hop GUI pipeline execution flow and existing metadata or cache handling, then clarify checkpoint storage, row limits, resume selection, and invalidation behavior; done means the development-only checkpoint workflow is implemented and verified without creating test-pipeline metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
developer-experience, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.