apache / apache/beam

Python SDK, DataflowRunner streaming job stops without error.

Open
#19,307 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Hello, hopefully I'm reporting this in the right location -

 

I have a simple dataflow job that reads from Pub/Sub, does a few transforms, and writes to Redis (in another PTransform at the moment, since no Python I/O connector). It runs perfectly well for several days, then the elements just stop being processed. If I update the job, or stop and start, it carries on again for a few more days. I believe I am not the only one to have run into this issue, as I found mention on [Stack Overflow](https://stackoverflow.com/questions/53610876/dataflow-stops-streaming-to-bigquery-without-errors). The only indication of workers quitting I've discovered so far are these info logs:

 
```

[0129/074659:INFO:update_manager-inl.h(52)] ChromeOSPolicy::UpdateCheckAllowed: START
[0129/074659:WARNING:evaluation_context-inl.h(43)]
Error reading Variable update_disabled: "No value set for update_disabled"
[0129/074659:WARNING:evaluation_context-inl.h(43)]
Error reading Variable release_channel_delegated: "No value set for release_channel_delegated"
[0129/074659:INFO:chromeos_policy.cc(317)]
Periodic check interval not satisfied, blocking until 1/29/2019 8:26:37 GMT
[0129/074659:INFO:update_manager-inl.h(74)]
ChromeOSPolicy::UpdateCheckAllowed: END

```

I have also filed a Dataflow bug, as I'm not sure where the problem lies. Any help resolving this would be very welcome, and If I can provide anything further, please let me know. Thank you!

 

 

Imported from Jira [BEAM-6576](https://issues.apache.org/jira/browse/BEAM-6576). Original Jira may contain additional context.
Reported by: dsnowberger.

Contributor guide

Open the contributing guide

Research direction

Start with the reported Python SDK DataflowRunner streaming job that reads from Pub/Sub and writes to Redis, then review the worker logs and the linked Stack Overflow and Jira BEAM-6576 context. Done means determining why processing stops after several days without an error and documenting or validating a resolution that survives restarting or updating the job.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.