TimelyDataflow / TimelyDataflow/differential-dataflow

Trace import in an iterative context appears to create downgrade loop

Open
#598 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3k
Forks
211
Avg merge
10h 42m
Merged PRs (30d)
34

Description

Hi,

I'm a bit stuck trying to use a shared trace in an iterative scope. I'm trying to emulate a database transaction, where you have some existing data, updates in a transaction, and if these updates pass some validation they should become part of the "existing data". I have run into an issue where the trace import operator seems to get stuck in a downgrade loop.

To create a minimal repro, I have a program where I have a set of pre-existing values (simple string values "a", "b", "c"), and I want to be able to submit updates to these values. I submit 2 transactions:

  • Transaction 1 adds "c" - invalid because "c" already exists
  • Transaction 2 adds "d" - valid

The transaction updates enter an iterative context, semi-join with existing committed values to find "violations", and then set a validated_tx_var variable with the valid transaction ID's (2 in this case). The "committed" values are defined as a values input collection, plus updates semi-joined with validated_tx_var.

If I turn the committed values into a trace and then use that to find violations, it appears to create a downgrade loop. I added a no-op inspect_frontier operator that prints out the input frontier (without holding onto any capabilities) and it prints out (0,1), (0,2), ... (0,n) etc. The code inside TraceAgent::import_core appears to be repeatedly calling capabilities.downgrade(&frontier.borrow()[..]); with TraceReplayInstruction::Frontier as the matched instruction.

Here is the code: https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=26347c68458ecf8c175f45cf77a313c6. You can toggle the bool on line 29 to see broken vs expected behaviour. Hopefully I've just missed something in the rules for using traces :)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the linked Rust playground repro and toggle the boolean on line 29 to compare the broken and expected behavior. Then inspect TraceAgent::import_core, especially the capabilities.downgrade call matched by TraceReplayInstruction::Frontier. Done means the trace import no longer enters the downgrade loop and the valid transaction behavior matches the expected result.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.