apache / apache/beam

Initial size estimation fails for mobile gaming examples for DataflowRunner when run in Windows

Open
#18,247 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

This happens for pipelines user_score.py and hourly_team_score.py

Jobs complete successfully but initial size estimation fails with following error.

INFO:root:Could not estimate size of source due to an exception
: Traceback (most recent call last):
File "C:\Users\chamikara\pythontest1\rc4_test\env_rc4_1\lib\site-packages\apache_beam\runners\dataflow\dataflow_runner
.py", line 554, in run_Read
transform.source.estimate_size())
File "C:\Users\chamikara\pythontest1\rc4_test\env_rc4_1\lib\site-packages\apache_beam\internal\gcp\json_value.py", lin
e 59, in get_typed_value_descriptor
raise TypeError('Cannot get a type descriptor for %s.' % repr(obj))
TypeError: Cannot get a type descriptor for 23899840340L.

Seems to be due to a combination of Windows returning Long for type name [1] and SDK not handling long type at [2].

[1] http://stackoverflow.com/questions/22513445/python-handles-long-ints-differently-on-windows-and-unix

[2] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/internal/gcp/json_value.py#L35

cc: [~altay]

Imported from Jira [BEAM-2294](https://issues.apache.org/jira/browse/BEAM-2294). Original Jira may contain additional context.
Reported by: chamikara.

Contributor guide

Open the contributing guide

Research direction

Inspect the Python DataflowRunner path in dataflow_runner.py around run_Read and the type handling in internal/gcp/json_value.py, using user_score.py and hourly_team_score.py as reproductions. Check how Windows represents the estimated source size and verify that both examples complete initial size estimation without the reported TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.