Profile and improve performance for the local FnApiRunner
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
The FnApiRunner is undergoing a series of changes to support streaming. These changes are altering its execution significantly, and may introduce inefficiencies.
This project has the following deliverables:
* A report with results from profiling the execution of a pipeline, and finding hotspots, and inefficiencies
* Code improvements to speed up the execution of the FnApiRunner
* Improvements to the [FnApiRunner manual](https://docs.google.com/document/d/18-AhNn1VG134ti1A7A__BtLQQg-99VmjRoNlcWg-8Qg/edit#) to instruct others on how to do profiling.
Tools that you may need to use:
* Benchmarks for the Python direct runner in [http://metrics.beam.apache.org](http://metrics.beam.apache.org/)
* The FnApiRunner and TestStream microbenchmarks in sdks/python/apache_beam/tools/
* A profiling tool like: [https://pypi.org/project/flameprof/](https://pypi.org/project/flameprof/) or [https://github.com/uber-archive/pyflame](https://github.com/uber-archive/pyflame)
* Google Docs
Contact Pablo in [dev@beam.apache.org|mailto:dev@beam.apache.org] to ask questions about this project.
Imported from Jira [BEAM-11869](https://issues.apache.org/jira/browse/BEAM-11869). Original Jira may contain additional context.
Reported by: pabloem.
Subtask of issue #21987
Contributor guide
Assessment
This issue has not been assessed yet.