apache / apache/beam

Python SDK should stage only up-to-date versions of pipeline dependencies defined by requirements file.

Open
#19,968 0 comments 0 reactions 0 assignees View on GitHub
bug core good first issue P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When staging pipeline dependencies specified in requirements.txt file and transitive deps, we stage entire content of requirements-cache directory on user's machine, which may include dependencies that were cached a while ago and are no longer needed. See [1] for discussion and some ideas how we can avoid staging packages we don't need.

[1] https://lists.apache.org/thread.html/f99820489026a46160505ea9a4c8c4a0eabe8649143c1ee22b7088aa%40%3Cdev.beam.apache.org%3E

Imported from Jira [BEAM-8900](https://issues.apache.org/jira/browse/BEAM-8900). Original Jira may contain additional context.
Reported by: tvalentyn.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.