Switch from `python_packages` to `python_requirements`
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 9.5k
- Forks
- 696
- Avg merge
- 7d 19h
- Merged PRs (30d)
- 2
Description
The design is outlined in this comment:
- support the requirements.txt format via
python_requirementsin cog.yaml. This exists today but is currently undocumented. - continue to support
python_packagesin cog.yaml, but remove documentation for it. - leave the door open for other dependency file types later (pipfile.lock, etc), but only support requirements.txt for now
- update
python_requirementsparsing behavior:- if an unqualified version of pytorch and tensorflow is specified (like
1.11.0), do magic to find the right version (torch==1.11.0+cpu,torch==1.11.0+cu113), and log it so the user knows this magic is happening. - if a fixed version of pytorch or tensorflow is specified by the user with extra qualifiers like
torch==1.11.0+cu113ortorch==1.11.0+cpu, don't do any magic and install that exact version, and tell the user we're not doing any magic. - if a version range (
package>=0.2,<0.3) is specified in requirements.txt (for any package), let pip handle the version resolution, but log a warning to the user recommending that versions in requirements.txt should be pinned for the sake of reproducibility.
- if an unqualified version of pytorch and tensorflow is specified (like
Todo
- #714
- Define Python requirements with
requirements.txtandpython_requirements - Make
python_packagesgenerate arequirements.txtfor backwards compatibility - Parse
python_requirementsto determine versions
- Define Python requirements with
- Tell user that magic is happening
- Display warnings if torch or tensorflow version can't resolved
- Documentation
- Update docs and examples
Design process
We should decide between python_requirements and python_packages and make it work properly.
There is some background here that is not written down anywhere that needs writing down. tl;dr python_requirements doesn't work properly, python_packages is not ideal in various ways.
Potential designs
- Leave it how it is, remove
python_requirements. The main downside of this is Python requirements are either in a non-standard location or duplicated. - Switch to
python_requirements, and use a simple parser to determinetorchandtensorflowversions. The main downside of this is that it is not clear to the user that Cog is rewriting versions. - Switch to
python_requirements, but split outtorchandtensorflowas separate top-level options to make it clear they do some special sauce behind the scenes. Iftorch/tensorflowis detected inpython_requirements, then it ignores them, warns the user, or something else sensible.
User data
- Streamlit have learned that users normally already have some way of defining their dependencies (pip, Poetry, Conda, etc) and are unwilling to switch to a different method. So, they read dependencies from wherever the user has already defined them.
- Python dependencies should automatically install when you open a GitHub codespaces. They don't do that if they're inside
python_packages, but they do if they're in a plainrequirements.txt
(Consider this a wiki and please edit! Edited by @bfirsh, ...)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked design comment and inspect the existing python_requirements handling described in the issue. Trace how cog.yaml and requirements.txt dependencies are parsed, then verify the requested logging, warnings, exact-version behavior, and documentation updates for the listed PyTorch and TensorFlow cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100