apache / apache/beam

[Feature Request][Go SDK]: Support Slowly Changing Side Input Pattern

Open
#23,106 8 comments 2 reactions 0 assignees View on GitHub
go new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

Requires implementing a Periodic Impulse and Periodic Sequence transform for occasional re-reads of some data, in a large window.

Requires using those re-reads as a side input.

Requires allowing the map_windows urn: https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L296

So that the SDK can map windows from one to another, this needs to be added to the big switch in exec/translate.go.

Currently, trying to manually use this pattern leads to a panic on execution when a runner makes the mapping request:

https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/translate.go#L734

### Issue Priority

Priority: 1

### Issue Component

Component: sdk-go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.