apache / apache/beam

DirectRunner unnecessarily re-scheules tasks after exceptions

Open
#18,141 2 comments 0 reactions 1 assignee Claimed by @tariq-hasan View on GitHub
bug core good first issue P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Seems like DirectRunner keeps scheduling tasks when exceptions occur when reading BigQuery results (and possibly in other cases).

I verified that rescheduling is not coming from BigQuery. AFAIKT a _MonitorTask that gets added at following location does not get removed properly when an exception is thrown.
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/direct/executor.py#L361

To reproduce:
(1) Raise a 'ValueError' at the beginning of method BigQueryWrapper.convert_row_to_dict at following location.
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/bigquery.py#L1061
(2) Setup Python SDK and run bigquery_tornadoes with DirectRunner.
python -m apache_beam.examples.cookbook.bigquery_tornadoes \--output \--project

Imported from Jira [BEAM-1462](https://issues.apache.org/jira/browse/BEAM-1462). Original Jira may contain additional context.
Reported by: chamikara.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.