DirectRunner GameStats fails, ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']

Open

#19,540 opened on Jun 4, 2022 0 comments 0 reactions 0 assignees View on GitHub

direct gearpump improvement P3 runners

Dominant language: Java
Stars: 8.7k
Forks: 4.7k
Avg merge: 1d 20h
Merged PRs (30d): 196

Description

GameStats example is failing with following logs
```

*****************************************************
* Running GameStats with DirectRunner
*****************************************************
INFO:root:Missing
pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1137:
BeamDeprecationWarning: options is deprecated since First stable release. References to .options
will not be supported
method_to_use = self._compute_method(p, p.options)
INFO:root:Running pipeline
with DirectRunner.
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing
access_token
INFO:root:Created table google.com:clouddfe.goenka_python_validations.game_stats_teams
with schema , ,
, ]>. Result: , , ,
]>
selfLink: u'https://www.googleapis.com/bigquery/v2/projects/google.com:clouddfe/datasets/goenka_python_validations/tables/game_stats_teams'
tableReference:

type: u'TABLE'>.
INFO:oauth2client.transport:Attempting refresh to obtain initial
access_token
INFO:oauth2client.client:Refreshing access_token
ERROR:root:Exception at bundle , due to an exception.
Traceback (most recent call last):
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 343, in call
finish_state)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 380, in attempt_call
evaluator.process_element(value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py",
line 633, in process_element
self.runner.process(element)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 753, in process
self._reraise_augmented(exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 807, in _reraise_augmented
raise_with_traceback(new_exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 751, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 564, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 571, in _invoke_process_per_window
side_inputs = [si[window] for si in self.side_inputs]
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
line 65, in __getitem__
_FilteringIterable(self._iterable, target_window), self._view_options)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 346, in _from_runtime_iterable
return options['data'].view_fn(it)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 323, in
lambda iterable: from_runtime_iterable(iterable, view_options))
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 433, in _from_runtime_iterable
len(head), str(head[0]), str(head[1])))
ValueError: PCollection
of size 2 with more than one element accessed as a singleton view. First two elements encountered are
"181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']

ERROR:root:Exception
at bundle , due to an exception.
Traceback
(most recent call last):
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 343, in call
finish_state)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 380, in attempt_call
evaluator.process_element(value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py",
line 633, in process_element
self.runner.process(element)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 753, in process
self._reraise_augmented(exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 807, in _reraise_augmented
raise_with_traceback(new_exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 751, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 564, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 571, in _invoke_process_per_window
side_inputs = [si[window] for si in self.side_inputs]
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
line 65, in __getitem__
_FilteringIterable(self._iterable, target_window), self._view_options)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 346, in _from_runtime_iterable
return options['data'].view_fn(it)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 323, in
lambda iterable: from_runtime_iterable(iterable, view_options))
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 433, in _from_runtime_iterable
len(head), str(head[0]), str(head[1])))
ValueError: PCollection
of size 2 with more than one element accessed as a singleton view. First two elements encountered are
"181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']

ERROR:root:Giving
up after 4 attempts.
WARNING:root:A task failed with exception: PCollection of size 2 with more than
one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1".
[while running 'CalculateSpammyUsers/ProcessAndFilter']
INFO:oauth2client.transport:Attempting refresh
to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
Traceback (most recent
call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname,
loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/lib/python2.7/site-packages/apache_beam/examples/complete/game/game_stats.py",
line 399, in
run()
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/lib/python2.7/site-packages/apache_beam/examples/complete/game/game_stats.py",
line 393, in run
}, options.view_as(GoogleCloudOptions).project))
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 426, in __exit__
self.run().wait_until_finish()
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
line 430, in wait_until_finish
self._executor.await_completion()
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 400, in await_completion
self._executor.await_completion()
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 446, in await_completion
raise_(t, v, tb)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 343, in call
finish_state)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 380, in attempt_call
evaluator.process_element(value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py",
line 633, in process_element
self.runner.process(element)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 753, in process
self._reraise_augmented(exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 807, in _reraise_augmented
raise_with_traceback(new_exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 751, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 564, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 571, in _invoke_process_per_window
side_inputs = [si[window] for si in self.side_inputs]
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
line 65, in __getitem__
_FilteringIterable(self._iterable, target_window), self._view_options)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 346, in _from_runtime_iterable
return options['data'].view_fn(it)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 323, in
lambda iterable: from_runtime_iterable(iterable, view_options))
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 433, in _from_runtime_iterable
len(head), str(head[0]), str(head[1])))
ValueError: PCollection
of size 2 with more than one element accessed as a singleton view. First two elements encountered are
"181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']

```

Imported from Jira [BEAM-7471](https://issues.apache.org/jira/browse/BEAM-7471). Original Jira may contain additional context.
Reported by: angoenka.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

DirectRunner GameStats fails, ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']

Description

Contributor guide

Assessment

Get new issues in your inbox