openedx / openedx/openedx-platform
Improve edx-platform requirements file structure
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.2k
- Forks
- 4.4k
- Avg merge
- 6d 18h
- Merged PRs (30d)
- 42
Description
edx-platform's requirements files pretty closely follow the recommendations in OEP-18. But as Kyle McCormick pointed out recently, the pip-tools README has since been updated to recommend something a little different. Try making the following changes:
- Benchmark the performance of make upgrade and make requirements before making any changes.
- Instead of basing each .in file on the previous fully pinned output file via -r, use -c instead. This should result in a requirements file that has just the dependencies needed for the packages listed in the file currently being processed, but at versions compatible with the previous layer of dependencies.
- Rename the output files generated from an input file that builds on top of a previous layer with a _layer suffix, like testing_layer.txt.
- Introduce new files with the original output names that include all the dependencies the previous ones did; for example, the new testing.txt would consist of the lines -r base.txt, -r coverage.txt, and -r testing_layer.txt.
- Benchmark the performance of make upgrade and make requirements after the changes.
What we hope to gain from this change are the following:
- Faster
make upgradeperformance by not having to process as many dependencies for each layer - Better ease of reviewing dependency updates, since most dependency versions will only appear in a single requirements file, whereas many currently appear in 3 or more.
In the process, we will lose having a single file (development.txt) that lists essentially all of the packages used by edx-platform. But we rarely use this, and we could easily reproduce it by running pip-compile on the new development.txt.
If the results are good, write new tickets for the following:
- An update to OEP-18 to recommend this new structure for requirements files
- Updating all repos owned by arch-bom to comply with the new structure (these could be either BOM or ARCHBOM tickets)
- Communicating the change to the rest of the organization
Copied description from this private JIRA issue: https://2u-internal.atlassian.net/browse/BOM-2483
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating edx-platform's layered requirements files and the Make targets for make upgrade and make requirements. Benchmark both commands, then compare the current structure with the pip-tools layered-requirements workflow and apply the requested file and naming changes. Done means the new structure works and post-change benchmarks are recorded; follow-up tickets are separate work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, tooling
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100