Python SDK, DataflowRunner streaming job stops without error.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Hello, hopefully I'm reporting this in the right location -
I have a simple dataflow job that reads from Pub/Sub, does a few transforms, and writes to Redis (in another PTransform at the moment, since no Python I/O connector). It runs perfectly well for several days, then the elements just stop being processed. If I update the job, or stop and start, it carries on again for a few more days. I believe I am not the only one to have run into this issue, as I found mention on [Stack Overflow](https://stackoverflow.com/questions/53610876/dataflow-stops-streaming-to-bigquery-without-errors). The only indication of workers quitting I've discovered so far are these info logs:
```
[0129/074659:INFO:update_manager-inl.h(52)] ChromeOSPolicy::UpdateCheckAllowed: START
[0129/074659:WARNING:evaluation_context-inl.h(43)]
Error reading Variable update_disabled: "No value set for update_disabled"
[0129/074659:WARNING:evaluation_context-inl.h(43)]
Error reading Variable release_channel_delegated: "No value set for release_channel_delegated"
[0129/074659:INFO:chromeos_policy.cc(317)]
Periodic check interval not satisfied, blocking until 1/29/2019 8:26:37 GMT
[0129/074659:INFO:update_manager-inl.h(74)]
ChromeOSPolicy::UpdateCheckAllowed: END
```
I have also filed a Dataflow bug, as I'm not sure where the problem lies. Any help resolving this would be very welcome, and If I can provide anything further, please let me know. Thank you!
Imported from Jira [BEAM-6576](https://issues.apache.org/jira/browse/BEAM-6576). Original Jira may contain additional context.
Reported by: dsnowberger.
Contributor guide
Research direction
Start with the reported Python SDK DataflowRunner streaming job that reads from Pub/Sub and writes to Redis, then review the worker logs and the linked Stack Overflow and Jira BEAM-6576 context. Done means determining why processing stops after several days without an error and documenting or validating a resolution that survives restarting or updating the job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100