apache / apache/beam

Versions after `2.28.0` fail to infer grouping decoders after a date is selected from a data structure

Open
#21,161 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The below code throws this type error on the effected versions, and merely works as expected on 2.28.0:

`TypeError: Unable to deterministically encode '2021-11-02' of type '', please provide a type hint for the input of 'GroupByKey' [while running 'Create/Map(decode)']`

```

import typing
from datetime import date
import apache_beam as beam
from apache_beam.testing.test_pipeline
import TestPipeline

with TestPipeline() as pipeline:
today = date.today()
results = (
pipeline
|
beam.Create([(1, { 'd': today }), (1, { 'd': today })])
| beam.MapTuple(lambda i, d: (d['d'], i))
# <-- this step only requires output type hints on versions after 2.28.0, and only if the date is being
"projected" from some other data structure
| beam.CombinePerKey(sum) # <-- if this aggregation is
removed, the pipeline also works without error
)

results | beam.Map(print)

```

This stackoverflow issue is having the same problem:
https://stackoverflow.com/questions/69409693/how-do-i-use-a-datetime-date-value-in-apache-beam-groupby

It's possible to fix the errors by registering a `DateCoder` and adding output type hints to the projection `MapTuple` step, but since this isn't necessary in other situations and versions, it seems this is a bug. Our production pipelines will need to add many of these tedious type hints in order to work properly, so we're effectively blocked from upgrading to the newest version.

Imported from Jira [BEAM-13166](https://issues.apache.org/jira/browse/BEAM-13166). Original Jira may contain additional context.
Reported by: blaine.hansen@marketdial.com.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.