Versions after `2.28.0` fail to infer grouping decoders after a date is selected from a data structure
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
The below code throws this type error on the effected versions, and merely works as expected on 2.28.0:
`TypeError: Unable to deterministically encode '2021-11-02' of type '', please provide a type hint for the input of 'GroupByKey' [while running 'Create/Map(decode)']`
```
import typing
from datetime import date
import apache_beam as beam
from apache_beam.testing.test_pipeline
import TestPipeline
with TestPipeline() as pipeline:
today = date.today()
results = (
pipeline
|
beam.Create([(1, { 'd': today }), (1, { 'd': today })])
| beam.MapTuple(lambda i, d: (d['d'], i))
# <-- this step only requires output type hints on versions after 2.28.0, and only if the date is being
"projected" from some other data structure
| beam.CombinePerKey(sum) # <-- if this aggregation is
removed, the pipeline also works without error
)
results | beam.Map(print)
```
This stackoverflow issue is having the same problem:
https://stackoverflow.com/questions/69409693/how-do-i-use-a-datetime-date-value-in-apache-beam-groupby
It's possible to fix the errors by registering a `DateCoder` and adding output type hints to the projection `MapTuple` step, but since this isn't necessary in other situations and versions, it seems this is a bug. Our production pipelines will need to add many of these tedious type hints in order to work properly, so we're effectively blocked from upgrading to the newest version.
Imported from Jira [BEAM-13166](https://issues.apache.org/jira/browse/BEAM-13166). Original Jira may contain additional context.
Reported by: blaine.hansen@marketdial.com.
Contributor guide
Assessment
This issue has not been assessed yet.