apache / apache/beam

Consider not pinning all python dependencies, or moving them to requirements.txt

Open
#18,542 0 comments 0 reactions 0 assignees View on GitHub
build P3 wish
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Currently all python dependencies are [pinned or capped](https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97)

While there's a good argument for supplying a `requirements.txt` with well tested dependencies, having them specified in `setup.py` forces them to an exact state on each install of Beam. This makes using Beam in any environment with other libraries nigh on impossible.

This is particularly severe for the `gcp` dependencies, where we have libraries that won't work with an older version (but Beam _does_ work with an newer version). We have to do a bunch of gymnastics to get the correct versions installed because of this. Unfortunately, airflow repeats this practice and conflicts on a number of dependencies, adding further complication (but, again there is no real conflict).

I haven't seen this practice outside of the Apache & Google ecosystem - for example no libraries in numerical python do this. Here's a [discussion on SO](https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions)

Imported from Jira [BEAM-3106](https://issues.apache.org/jira/browse/BEAM-3106). Original Jira may contain additional context.
Reported by: maximilianr.

Contributor guide

Open the contributing guide

Research direction

Start with the dependency declarations in sdks/python/setup.py around line 97 and review how Python dependencies are installed. Compare the proposed setup.py and requirements.txt approaches, especially for the gcp dependencies and compatibility with other libraries. Done means Beam installation no longer unnecessarily forces dependency versions while retaining a documented, tested dependency set.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.