TimelyDataflow / TimelyDataflow/differential-dataflow
Trace import in an iterative context appears to create downgrade loop
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
Hi,
I'm a bit stuck trying to use a shared trace in an iterative scope. I'm trying to emulate a database transaction, where you have some existing data, updates in a transaction, and if these updates pass some validation they should become part of the "existing data". I have run into an issue where the trace import operator seems to get stuck in a downgrade loop.
To create a minimal repro, I have a program where I have a set of pre-existing values (simple string values "a", "b", "c"), and I want to be able to submit updates to these values. I submit 2 transactions:
- Transaction 1 adds
"c"- invalid because "c" already exists - Transaction 2 adds
"d"- valid
The transaction updates enter an iterative context, semi-join with existing committed values to find "violations", and then set a validated_tx_var variable with the valid transaction ID's (2 in this case). The "committed" values are defined as a values input collection, plus updates semi-joined with validated_tx_var.
If I turn the committed values into a trace and then use that to find violations, it appears to create a downgrade loop. I added a no-op inspect_frontier operator that prints out the input frontier (without holding onto any capabilities) and it prints out (0,1), (0,2), ... (0,n) etc. The code inside TraceAgent::import_core appears to be repeatedly calling capabilities.downgrade(&frontier.borrow()[..]); with TraceReplayInstruction::Frontier as the matched instruction.
Here is the code: https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=26347c68458ecf8c175f45cf77a313c6. You can toggle the bool on line 29 to see broken vs expected behaviour. Hopefully I've just missed something in the rules for using traces :)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the linked Rust playground repro and toggle the boolean on line 29 to compare the broken and expected behavior. Then inspect TraceAgent::import_core, especially the capabilities.downgrade call matched by TraceReplayInstruction::Frontier. Done means the trace import no longer enters the downgrade loop and the valid transaction behavior matches the expected result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100