apache / apache/beam

[Bug]: PickleCoder/_MemoizingPickleCoder.as_deterministic_coder() raises TypeError

Open
#39,942 1 comment 0 reactions 1 assignee Claimed by @AmirF194 View on GitHub
P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

`PickleCoder.as_deterministic_coder()` and `_MemoizingPickleCoder.as_deterministic_coder()` raise `TypeError` on every call instead of returning a working deterministic coder.

Both methods do:

```python
def as_deterministic_coder(self, step_label, error_message=None):
return FastPrimitivesCoder(self, requires_deterministic=step_label)
```

`FastPrimitivesCoder.__init__(self, fallback_coder=PickleCoder())` has never accepted a `requires_deterministic` argument, so this raises:

```
TypeError: FastPrimitivesCoder.__init__() got an unexpected keyword argument 'requires_deterministic'
```

Repro on current `master`:

```python
from apache_beam.coders.coders import PickleCoder
PickleCoder().as_deterministic_coder('label')
# TypeError: FastPrimitivesCoder.__init__() got an unexpected keyword argument 'requires_deterministic'
```

Root cause: commit 9be70c9861 (2021-03-08) changed `PickleCoder.as_deterministic_coder` from the working `DeterministicFastPrimitivesCoder(self, step_label)` to the broken `FastPrimitivesCoder(self, requires_deterministic=...)` in the same diff that added a `requires_deterministic` kwarg to a different class, `coder_impl.FastPrimitivesCoderImpl` (the Cython impl, not the `coders.FastPrimitivesCoder` wrapper referenced here). `_MemoizingPickleCoder` (added later) copied the same already-broken line.

Any caller that resolves to one of these coders and asks for a deterministic version hits this, for example `GroupByEncryptedKey.expand()` in `apache_beam/transforms/util.py`, which wraps the call in `except ValueError` and does not catch the `TypeError`, so it crashes instead of logging the intended non-deterministic-key warning.

This is a narrower, separate defect from #28558's own headline claim (which is about `pickle_library` being ignored, and was addressed by @tvalentyn as working as intended for `PickleCoder`). @tvalentyn identified this specific `as_deterministic_coder` bug in a comment on that thread in 2023 and invited a PR for it, but #28558 itself is now closed and its title does not match this fix, so filing it here as its own issue.

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- Component: Python SDK

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.