mesonbuild / mesonbuild/meson-python
Dynamic control over ninja parallelism, or generic dynamic features
Nessuno ha ancora preso questa issue.
- Lingua principale
- Python
- Stelle
- 180
- Fork
- 93
- Merge medio
- 2g 7h
- PR unite (30g)
- 18
Descrizione
First the specific feature we need in SciPy: set the number of parallel compile jobs for `ninja` to equal the number of available physical cores (`n_phys`). We have a number of reports about the `ninja` default setting, which is `2*n_phys + 2`, either giving out of memory errors or even crashing the build machine outright:
- https://github.com/scipy/scipy/issues/17941 is a report for a Chromebook running Debian (with a very good description of the problem and background info)
- https://github.com/scipy/scipy/issues/18443 reports the issue on multiple Windows machines, with a regular 8-core CPU and 16 GB RAM
- https://github.com/ninja-build/ninja/issues/2187 is a/the relevant upstream issue
For local development this was not difficult to address (see https://github.com/scipy/scipy/pull/18451), because SciPy has a developer interface where we can run custom code before invoking Meson. However, for `pip install` & co, there is no way to do that currently. So a user who may be installing some random package that happens to depend on SciPy may get crash or hang that we can't easily prevent. Hardcoding, say, `-j4` is not great - there is no setting that works on low-end machines without throwing away lots of performance on high-end machines. The optimal settings seems to always be close to the number of physical cores.
So, ideally there would be a hook that `meson-python` exposes to package authors which can run arbitrary Python code to then set `compile-args` with.
I also see a more general pattern here, with gh-159 being another example of the need to execute code first and then set some build-related property (the package version in that case). It's a very similar case, and rather than a specific solution for each of these two problems, we may consider a general mechanism to deal with this kind of thing. Maybe a single Python function to execute, which must return a dict containing a pre-determined set of keys, including `version`, `compile-args` and the other `*-args` knobs.
Thoughts?
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Direzione di ricerca
Inizia con l’interfaccia per sviluppatori esistente di meson-python e il SciPy PR 18451 collegato, quindi confronta il caso d’uso della versione gh-159 e l’upstream ninja issue. Il lavoro sarà completato quando sarà stato deciso e documentato un meccanismo per eseguire Python definito dal pacchetto prima del build e impostare proprietà supportate come compile-args e version.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- build-system, tooling
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 25/100