openedx / openedx/openedx-platform
Prevent inconsistent constraints in make upgrade
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.2k
- Forks
- 4.4k
- Avg merge
- 6d 18h
- Merged PRs (30d)
- 42
Description
Our usual approach to layered dependencies (particularly in edx-platform) with incremental dependency-solving has a flaw where one layer can produce a solution that is then incompatible with a later layer.
What we should do instead is to arrive at a global solution across all packages in all layers, then use that to build layer lockfiles separately. https://github.com/openedx/edx-platform/compare/timmc/megalock contains an attempt at implementing this.
- In this approach, all of the .in files are separate layers, with no reference to each other or even to the constraints file.
lock-all.inreferences all of them as well as the constraints file, and is compiled tolock-all.txt(a global solution file)use-lock.inis a piece of indirection that just says "use lock-all.txt as a constraints file"- Each layer file is then compiled with use-lock.in (pip-compile accepts multiple inputs) to produce dependency solutions that
are all constraint-compatible with each other and that only contain the packages necessary for that layer. make dev-requirementsand similar then usepip-syncwith multiple inputs, a list of all the layers that are desired for that workflow. Layers can be safely combined because they were all produced against a global constraints file.
Expected effects:
make upgradeshould be more reliable- The relationship between layers would be simpler (as they would no longer have an inheritance relationship)
- Possible performance improvements; performance should be no worse, at least.
- A slow-to-update shared dependency in dev.in will hold back base.txt. In the current state, we might instead get a
make upgradeerror. It's not clear whether this would be a positive or a negative change. - Reviewing the output of an upgrade might become simpler, as dev.txt will not contain changes that only affect base.txt. People would need to ignore changes to lock-all.txt when reviewing, as those would be entirely redundant. Alternatively, the file could be automatically deleted at the end of compiling requirements, but it has value in debugging dependency issues since it would be the only place that showed all package relationships.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Makefile targets for make upgrade and make dev-requirements, then inspect the existing layered .in and lockfile inputs. Review the linked comparison for its lock-all.in, lock-all.txt, and use-lock.in approach. Done means all layers are generated from one global solution and can be safely combined with pip-sync.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100