Pipeline context should properly initialize componentid map when it is created from a proto.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
When we re-create context from proto, we don't initialize component_id_map with existing objects in various mappings: https://github.com/apache/beam/blob/b43952ca369418e6f0d6cc2d4c54f991d7aafb25/sdks/python/apache_beam/runners/pipeline_context.py#L184-L211. Therefore we cannot create new entities in the context without stepping over existing IDs.
Imported from Jira [BEAM-12084](https://issues.apache.org/jira/browse/BEAM-12084). Original Jira may contain additional context.
Reported by: tvalentyn.
Contributor guide
Research direction
Start in sdks/python/apache_beam/runners/pipeline_context.py at the proto reconstruction code around lines 184-211. Inspect how existing objects in the mappings are restored and how component_id_map is populated; done means newly created entities do not reuse IDs already represented in the reconstructed context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100