[Feature Request]: Dev Cache Mode / Step Checkpoints (development only)
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
Goal
Speed up development and debugging cycles in Apache Hop pipelines by reusing a step output as a “checkpoint”, avoiding reprocessing the entire upstream on every run.
Concept (proposal)
The user marks a step as a checkpoint.
Hop stores the step output (rows/dataset) in a local cache.
On later runs, Hop can skip upstream steps and start from the checkpoint, loading data from cache.
The user explicitly chooses:
Run from the beginning
Run from the last checkpoint
Note
Unlike a “test pipeline”, the idea is not to create dataset metadata. This is a development-only cache feature to speed up iteration, especially when upstream extraction (e.g., from a database) is slow. The user can define where to store the cache and how many rows to save. At runtime, they can run from the beginning or resume from the last checkpoint, without creating extra metadata or separate test pipelines.
### Issue Priority
Priority: 3
### Issue Component
Component: Hop Gui
Contributor guide
Research direction
No files, tests, or entry points are named in the issue. Start by locating the Hop GUI pipeline execution flow and existing metadata or cache handling, then clarify checkpoint storage, row limits, resume selection, and invalidation behavior; done means the development-only checkpoint workflow is implemented and verified without creating test-pipeline metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- developer-experience, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100