Consider not pinning all python dependencies, or moving them to requirements.txt
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently all python dependencies are [pinned or capped](https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97)
While there's a good argument for supplying a `requirements.txt` with well tested dependencies, having them specified in `setup.py` forces them to an exact state on each install of Beam. This makes using Beam in any environment with other libraries nigh on impossible.
This is particularly severe for the `gcp` dependencies, where we have libraries that won't work with an older version (but Beam _does_ work with an newer version). We have to do a bunch of gymnastics to get the correct versions installed because of this. Unfortunately, airflow repeats this practice and conflicts on a number of dependencies, adding further complication (but, again there is no real conflict).
I haven't seen this practice outside of the Apache & Google ecosystem - for example no libraries in numerical python do this. Here's a [discussion on SO](https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions)
Imported from Jira [BEAM-3106](https://issues.apache.org/jira/browse/BEAM-3106). Original Jira may contain additional context.
Reported by: maximilianr.
Contributor guide
Research direction
Start with the dependency declarations in sdks/python/setup.py around line 97 and review how Python dependencies are installed. Compare the proposed setup.py and requirements.txt approaches, especially for the gcp dependencies and compatibility with other libraries. Done means Beam installation no longer unnecessarily forces dependency versions while retaining a documented, tested dependency set.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100