Journal mismatch message swaps "previous" and "current" when the command types differ

Open
#96 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
rust

Research direction

Start with PopJournalEntry in src/vm/transitions/journal.rs and the CommandTypeMismatchError Display implementation in src/vm/errors.rs; compare the actual and expected fields with the call sites in src/service_protocol/encoding.rs and the replay paths. Add the proposed regression test for replaying a recorded Run while the handler writes output, and confirm the mismatch message identifies the recorded and current commands correctly.

Written by the indexing model from the issue text.

Description

When a replay finds a recorded command of a different type than the one the handler now issues, the journal mismatch error (570, RT0016) names the two sides the wrong way round. It says the previous execution recorded the handler's command, and that the current execution attempts the recorded one.

Reproduction

Python SDK restate-sdk 1.0.5 (shared core 7.0.3) against restate-server 1.7.10. A runnable reproduction is public at https://github.com/SciMigo/restate-durable-agent-demo: docker compose up -d, then python demo.py --agent naive --model drifted --wait 150. The recorded run is in observed/3-naive-drifted.txt.

A handler calls a model outside ctx.run, uses the answer to decide whether to call a tool, and runs the tool with ctx.run("get_weather", …). The process is killed after the tool call. On replay, the model call runs again, answers differently, and the handler returns instead of calling the tool.

Journal of the invocation (sys_journal):

 0  Command: Input
 1  Command: Run  get_weather
 2  Notification: Run
 3  Command: Sleep
 4  Notification: Sleep

Error on every retry (from the SDK log, and the Paused event in sys_journal_events):

[570 Journal mismatch] Found a mismatch between the code paths taken during the previous execution and the paths taken during this execution.
This typically happens when some parts of the code are non-deterministic.
 - The previous execution ran and recorded the following: 'handler return' (index '1')
 - The current execution attempts to perform the following: 'run'

Expected: the previous execution recorded 'run', and the current execution attempts 'handler return'. The Paused event's own metadata agrees with the handler side: "related_command_index": 1, "related_command_type": "Output".

Cause
  • Replay names the two sides. PopJournalEntry (src/vm/transitions/journal.rs) pops the recorded command and names it actual; the command the handler issues is expected.
  • Callers keep that order. When the types differ, RawMessage::decode_to (src/service_protocol/encoding.rs:80) builds CommandTypeMismatchError::new(command_index, self.0.message_type(), M::ty()): the recorded type as actual, the handler's as expected. The lazy/eager state replay paths in journal.rs construct it the same way.
  • The message reverses it. impl Display for CommandTypeMismatchError (src/vm/errors.rs) prints self.expected under "The previous execution ran and recorded" and self.actual under "The current execution attempts".

The same-type case (CommandMismatchError) prints a field diff and is not affected. main has the same code as the 7.0.3 release.

Why it matters

This message is where people start when debugging non-determinism. Reversed, it points at the wrong code change: it reads as if the old code returned and the new code added a step. The TypeScript, Rust, Go and Java SDKs embed the same core, so they likely show the same message; I have only observed it with Python.

Proposed fix

Swap the two arguments in the Display impl and document what each field holds, with a regression test that replays a recorded Run while the handler writes output. I've opened a PR with this change.

Dominant language
Rust
Stars
7
Forks
14
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from restatedev/sdk-shared-core

All issues in restatedev/sdk-shared-core

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.