apache / apache/beam

Resolve differences in beam:metric:element_count:v1 implementations

Open
#20,204 0 comments 0 reactions 0 assignees View on GitHub
bug go harness java P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The [element count](https://github.com/apache/beam/blob/d82d061aa303430f3d2853f397f3130fae6200cd/model/pipeline/src/main/proto/metrics.proto#L206) metric represents the number of elements within a PCollection and is interpreted differently across the Beam SDK versions.

In the [Java SDK](https://github.com/apache/beam/blob/d82d061aa303430f3d2853f397f3130fae6200cd/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java#L207) this represents the number of elements and includes how many windows those elements are in. This metric is incremented as soon as the element has been output.

In the [Python SDK](https://github.com/apache/beam/blame/bfd151aa4c3aad29f3aea6482212ff8543ded8d7/sdks/python/apache_beam/runners/worker/opcounters.py#L247) this represents the number of elements and doesn't include how many windows those elements are in. The metric is also only incremented after the element has finished processing.

The [Go SDK](https://github.com/apache/beam/blob/7097850daa46674b88425a124bc442fc8ce0dcb8/sdks/go/pkg/beam/core/runtime/exec/datasource.go#L260) does the same thing as Python.

Traditionally in Dataflow this has always been the exploded window element count and the counter is incremented as soon as the element is output.

Imported from Jira [BEAM-9934](https://issues.apache.org/jira/browse/BEAM-9934). Original Jira may contain additional context.
Reported by: lcwik.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.