openedx / openedx/openedx-platform

Prevent inconsistent constraints in make upgrade

Open
#31,375 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.2k
Forks
4.4k
Avg merge
6d 18h
Merged PRs (30d)
42

Description

Our usual approach to layered dependencies (particularly in edx-platform) with incremental dependency-solving has a flaw where one layer can produce a solution that is then incompatible with a later layer.

What we should do instead is to arrive at a global solution across all packages in all layers, then use that to build layer lockfiles separately. https://github.com/openedx/edx-platform/compare/timmc/megalock contains an attempt at implementing this.

  • In this approach, all of the .in files are separate layers, with no reference to each other or even to the constraints file.
  • lock-all.in references all of them as well as the constraints file, and is compiled to lock-all.txt (a global solution file)
  • use-lock.in is a piece of indirection that just says "use lock-all.txt as a constraints file"
  • Each layer file is then compiled with use-lock.in (pip-compile accepts multiple inputs) to produce dependency solutions that
    are all constraint-compatible with each other and that only contain the packages necessary for that layer.
  • make dev-requirements and similar then use pip-sync with multiple inputs, a list of all the layers that are desired for that workflow. Layers can be safely combined because they were all produced against a global constraints file.

Expected effects:

  • make upgrade should be more reliable
  • The relationship between layers would be simpler (as they would no longer have an inheritance relationship)
  • Possible performance improvements; performance should be no worse, at least.
  • A slow-to-update shared dependency in dev.in will hold back base.txt. In the current state, we might instead get a make upgrade error. It's not clear whether this would be a positive or a negative change.
  • Reviewing the output of an upgrade might become simpler, as dev.txt will not contain changes that only affect base.txt. People would need to ignore changes to lock-all.txt when reviewing, as those would be entirely redundant. Alternatively, the file could be automatically deleted at the end of compiling requirements, but it has value in debugging dependency issues since it would be the only place that showed all package relationships.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Makefile targets for make upgrade and make dev-requirements, then inspect the existing layered .in and lockfile inputs. Review the linked comparison for its lock-all.in, lock-all.txt, and use-lock.in approach. Done means all layers are generated from one global solution and can be safely combined with pip-sync.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.