apache / apache/beam

Artifact stager should validate the manifest at pipeline submission time.

Open
#18,867 0 comments 0 reactions 0 assignees View on GitHub
core improvement java P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

One step of executing a Beam pipeline is staging pipeline dependencies to the runner, for Python SDK example see: [https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83](https://github.com/apache/beam/blob/1e2218aebf4902ca2f9107c885ff7ef0e1ef6eb8/sdks/python/apache_beam/runners/portability/stager.py#L83) and it's implementations. 

As a part of this process, we create a manifest of all staged artifacts. We should identify the requirements that constitute correctness of the manifest (for example, artifacts do not repeat twice, see also: https://github.com/apache/beam/blob/b0b7e3bf4941f874d360923ffd1d03e38befc589/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L122), and verify these requirements on the SDK side at pipeline submission, to fail faster.

 

Imported from Jira [BEAM-4407](https://issues.apache.org/jira/browse/BEAM-4407). Original Jira may contain additional context.
Reported by: tvalentyn.
This issue has child subcomponents which were not migrated over. See the original Jira for more information.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.