pathwaycom / pathwaycom/pathway
Introduce execution graph checksums
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 62.3k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
Is your feature request related to a problem? Please describe.
Pathway supports data persistence. To work, this feature essentially needs the guarantee that the graph hasn't been changed between the reruns of the program; otherwise, the rerunning program sees a checkpoint from a different graph, tried to parse it, sometimes succeeds, and then fails with an unclear error message.
To avoid spending time on debugging such problems, that may signify the framework bugs, but in reality correspond to an error caused by changing the graph between the two persistent runs, it would be good to calculate a hash/checksum of a Pathway execution graph, so that, at restart, the engine may identify the change in the graph and fail explicitly instead of trying to load data of the a priori incorrect format.
Describe the solution you'd like
Calculated checksum is added into the metadata of the persistent dump and is verified when the program loads it.
Compatibility
To make the change backward-compatible, the checksum can be written in a separate Key-Value pair in the selected backend without changing the format of the existing entries. On the start, if the pair is absent, the engine only calculates the checksum and saves it. In principle, if one wants to override the check (not recommended, and won't be documented), they can just drop the pair - perhaps may be useful in certain tests.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by locating Pathway's persistence and checkpoint-loading code, including selected-backend metadata handling, then trace execution-graph construction; done means a checksum is stored and verified on restart while absent metadata remains backward-compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100