apache / apache/beam

Sources using Metrics.counter fail with 'Can't pickle ... DelegatingCounter'

Open
#19,288 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

This fails under Python 2.7 but works fine under Python 3 – looks like an easy fix.

I am reading from a beam.io.utils.CountingSource:

`beam.io.Read(beam_io_utils.CountingSource(NUM_ROWS))`

This fails under python 2.7 with:

Python exception: Can't pickle : attribute lookup apache_beam.metrics.metric.DelegatingCounter failed

Looks like this is because DelegatingCounter is actually an inner class defined inside apache_beam.metrics.metric.Metric. Under Python 3 pickling can resolve it under its qualified name apache_beam.metrics.metric.Metric.DelegatingCounter, but not in Python 2. A fix would be to move it to a top-level class in that module.

This also applies to other sources which use Metrics.counter.

Imported from Jira [BEAM-6389](https://issues.apache.org/jira/browse/BEAM-6389). Original Jira may contain additional context.
Reported by: matthjw.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.