apache / apache/beam

DirectRunner does not update reference to currentRestriction when running in SDF

Open
#21,637 0 comments 0 reactions 0 assignees View on GitHub
bug direct P2 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I have an SDF implementation that looks like so:

 
```

class MyRestrictionTracker {
MyRestriction restriction;

currentRestriction() { return restriction;
}

tryClaim(MyPosition position) {
this.restriction = new MyRestriction(position)
}
}
```

I ran this on the DirectRunner, and the restriction would never advance: It would get stuck on the very first value.

I also ran this on DataflowRunner, and the problem did not exist there: This ran fine.

 

I was able to fix this on the DirectRunner (it works well on Dataflow as well) by changing the restriction to be mutable. Something like this:

 
```

class MyRestrictionTracker {
MyRestriction restriction;

currentRestriction() { return restriction;
}

tryClaim(MyPosition position) {
this.restriction.position = position;
}
}
```

This looks like an execution issue with SDF on DirectRunner: The DirectRunner is likely storing a reference to `currentRestriction()` and never updating it as it runs.

 

I'm happy to fix this on the DirectRunner - I would just like to find pointers : )

Imported from Jira [BEAM-14387](https://issues.apache.org/jira/browse/BEAM-14387). Original Jira may contain additional context.
Reported by: pabloem.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.