openedx / openedx/openedx-platform

Improve edx-platform requirements file structure

Open
#33,875 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.2k
Forks
4.4k
Avg merge
6d 18h
Merged PRs (30d)
42

Description

edx-platform's requirements files pretty closely follow the recommendations in OEP-18. But as Kyle McCormick pointed out recently, the pip-tools README has since been updated to recommend something a little different. Try making the following changes:

  1. Benchmark the performance of make upgrade and make requirements before making any changes.
  2. Instead of basing each .in file on the previous fully pinned output file via -r, use -c instead. This should result in a requirements file that has just the dependencies needed for the packages listed in the file currently being processed, but at versions compatible with the previous layer of dependencies.
  3. Rename the output files generated from an input file that builds on top of a previous layer with a _layer suffix, like testing_layer.txt.
  4. Introduce new files with the original output names that include all the dependencies the previous ones did; for example, the new testing.txt would consist of the lines -r base.txt, -r coverage.txt, and -r testing_layer.txt.
  5. Benchmark the performance of make upgrade and make requirements after the changes.

What we hope to gain from this change are the following:

  • Faster make upgrade performance by not having to process as many dependencies for each layer
  • Better ease of reviewing dependency updates, since most dependency versions will only appear in a single requirements file, whereas many currently appear in 3 or more.

In the process, we will lose having a single file (development.txt) that lists essentially all of the packages used by edx-platform. But we rarely use this, and we could easily reproduce it by running pip-compile on the new development.txt.

If the results are good, write new tickets for the following:

  • An update to OEP-18 to recommend this new structure for requirements files
  • Updating all repos owned by arch-bom to comply with the new structure (these could be either BOM or ARCHBOM tickets)
  • Communicating the change to the rest of the organization

Copied description from this private JIRA issue: https://2u-internal.atlassian.net/browse/BOM-2483

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating edx-platform's layered requirements files and the Make targets for make upgrade and make requirements. Benchmark both commands, then compare the current structure with the pip-tools layered-requirements workflow and apply the requested file and naming changes. Done means the new structure works and post-change benchmarks are recorded; follow-up tickets are separate work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.