apache / apache/beam

[Bug]: Example sql_taxi.py not working with DirectRunner

Open
#22,709 8 comments 0 reactions 0 assignees View on GitHub
bug examples P2 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

I am trying to run the example [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/sql_taxi.py](sdks/python/apache_beam/examples/sql_taxi.py) on my local machine. After some minor adaptations (I created a subscription to the "taxirides-realtime" topic and reference that instead of the topic itself), I expected it to work. But I get a NotImplementedError that indicates that SqlTransform is not implemented in DirectRunner.

The stacktrace I get:
```python
ERROR:apache_beam.runners.direct.executor:Exception at bundle , due to an exception.
Traceback (most recent call last):
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 370, in call
self.attempt_call(
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 400, in attempt_call
evaluator = self._transform_evaluator_registry.get_evaluator(
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 135, in get_evaluator
raise NotImplementedError(
NotImplementedError: Execution of [] not implemented in runner .

ERROR:apache_beam.runners.direct.executor:Exception at bundle , due to an exception.
Traceback (most recent call last):
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 370, in call
self.attempt_call(
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 400, in attempt_call
evaluator = self._transform_evaluator_registry.get_evaluator(
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 135, in get_evaluator
raise NotImplementedError(
NotImplementedError: Execution of [] not implemented in runner .

ERROR:apache_beam.runners.direct.executor:Exception at bundle , due to an exception.
Traceback (most recent call last):
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 370, in call
self.attempt_call(
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 400, in attempt_call
evaluator = self._transform_evaluator_registry.get_evaluator(
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 135, in get_evaluator
raise NotImplementedError(
NotImplementedError: Execution of [] not implemented in runner .

ERROR:apache_beam.runners.direct.executor:Exception at bundle , due to an exception.
Traceback (most recent call last):
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 370, in call
self.attempt_call(
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 400, in attempt_call
evaluator = self._transform_evaluator_registry.get_evaluator(
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 135, in get_evaluator
raise NotImplementedError(
NotImplementedError: Execution of [] not implemented in runner .

ERROR:apache_beam.runners.direct.executor:Giving up after 4 attempts.
WARNING:apache_beam.runners.direct.executor:A task failed with exception: Execution of [] not implemented in runner .
Traceback (most recent call last):
File "/Users/ture.friese/atl-dataflow-sql/sql_taxi.py", line 97, in
run(known_args.output_topic, pipeline_args)
File "/Users/ture.friese/atl-dataflow-sql/sql_taxi.py", line 46, in run
with beam.Pipeline(options=pipeline_options) as pipeline:
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/pipeline.py", line 598, in __exit__
self.result.wait_until_finish()
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/direct_runner.py", line 588, in wait_until_finish
self._executor.await_completion()
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 432, in await_completion
self._executor.await_completion()
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 480, in await_completion
raise update.exception
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 370, in call
self.attempt_call(
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/executor.py", line 400, in attempt_call
evaluator = self._transform_evaluator_registry.get_evaluator(
File "/Users/ture.friese/miniconda3/envs/atldfs/lib/python3.10/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 135, in get_evaluator
raise NotImplementedError(
NotImplementedError: Execution of [] not implemented in runner .
```

### Issue Priority

Priority: 2

### Issue Component

Component: examples-python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.