Python SDK should stage only up-to-date versions of pipeline dependencies defined by requirements file.
Open
bug
core
good first issue
P3
python
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When staging pipeline dependencies specified in requirements.txt file and transitive deps, we stage entire content of requirements-cache directory on user's machine, which may include dependencies that were cached a while ago and are no longer needed. See [1] for discussion and some ideas how we can avoid staging packages we don't need.
[1] https://lists.apache.org/thread.html/f99820489026a46160505ea9a4c8c4a0eabe8649143c1ee22b7088aa%40%3Cdev.beam.apache.org%3E
Imported from Jira [BEAM-8900](https://issues.apache.org/jira/browse/BEAM-8900). Original Jira may contain additional context.
Reported by: tvalentyn.
Contributor guide
Assessment
This issue has not been assessed yet.