apache / apache/beam

Typehint for Python CombineFn Accumulator

Open
#19,913 0 comments 0 reactions 0 assignees View on GitHub
core new feature P3 python types
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Feature request:
Allow users to specify a typehint for the accumulator type of CombineFn, either through a typehints decorator, or by using Python type annotations on the CombineFn methods, or through inheriting a CombineFn [generic type](https://docs.python.org/3/library/typing.html#user-defined-generic-types).

Benefits:
* Allow the user to [specify a more efficient coder](https://beam.apache.org/documentation/programming-guide/#data-encoding-and-type-safety) for accumulators if one is available. Currently users are able to do this for the input and output elements, but not the accumulators, which [always uses the fallback coder](https://github.com/apache/beam/blob/d664592ee761d45b0273edb027a18c6ed9340349/sdks/python/apache_beam/transforms/core.py#L859-L860).
* Allows typechecking of methods for CombineFn e.g. check that create_accumulator(), add_input() and merge_accumulators() returns accumulators. This provides better developer ergonomics as the CombineFnAPI is non-trivial.

Imported from Jira [BEAM-8519](https://issues.apache.org/jira/browse/BEAM-8519). Original Jira may contain additional context.
Reported by: myffical@gmail.com.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.