apache / apache/beam

Move Chicago Taxi Example to Python 3

Open
#19,973 0 comments 0 reactions 0 assignees View on GitHub
improvement P4 tests
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The Chicago Taxi Example[1] should be moved to the latest version of Python supported by Beam (currently it's Python 3.7).

At the moment, the following error occurs when running the benchmark on Python 3.7 (requires futher investigation):
```

Traceback (most recent call last):
File "preprocess.py", line 259, in
main()
File
"preprocess.py", line 254, in main
project=known_args.metric_reporting_project
File "preprocess.py",
line 155, in transform_data
('Analyze' >> tft_beam.AnalyzeDataset(preprocessing_fn)))
File "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/transforms/ptransform.py",
line 987, in __ror__
return self.transform.__ror__(pvalueish, self.label)
File "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/transforms/ptransform.py",
line 547, in __ror__
result = p.apply(self, pvalueish, label)
File "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/pipeline.py",
line 532, in apply
return self.apply(transform, pvalueish)
File "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/pipeline.py",
line 573, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)

File "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/runners/runner.py", line 193, in apply

return m(transform, input, options)
File "/Users/kamilwasilewski/proj/beam/sdks/python/apache_beam/runners/runner.py",
line 223, in apply_PTransform
return transform.expand(input)
File "/Users/kamilwasilewski/proj/beam/build/gradleenv/2022703441/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py",
line 825, in expand
input_metadata))
File "/Users/kamilwasilewski/proj/beam/build/gradleenv/2022703441/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py",
line 716, in expand
output_signature = self._preprocessing_fn(copied_inputs)
File "preprocess.py",
line 102, in preprocessing_fn
_fill_in_missing(inputs[key]),
KeyError: 'company'

```

[1] sdks/python/apache_beam/testing/benchmarks/chicago_taxi

Imported from Jira [BEAM-9154](https://issues.apache.org/jira/browse/BEAM-9154). Original Jira may contain additional context.
Reported by: kamilwu.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.