conda-forge / conda-forge/conda-forge.github.io

RFC: New compiler wrappers for handling C++ & Fortran modules

Open
#2,525 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
170
Forks
320
Avg merge
2d 10h
Merged PRs (30d)
5

Description

It's not every day that we need to modify something in our compiler setup, so I wanted to give this topic some more visibility.

## Situation

Both Fortran and C++ have a concept of so-called modules, which are addressing (among other things) a desire to avoid recompiling the same code over and over, without going as far as putting the code into an explicit library.

Fortran introduced these in F90 already, whereas C++ only did so in C++20 (and famously, the [transition](https://arewemodulesyet.org/) is slow-going, as support by compilers and build orchestrators is still work-in-progress).

One key aspect of modules is that -- in contrast to a full-fledged library -- the modules can only be consumed by the same compiler that produced them. This constraint is missing from our metdata.

In https://github.com/conda-forge/flang-activation-feedstock/issues/14, I came up with the following chain of considerations (for Fortran, but the same applies to C++)
1. Not every fortran project produces modules
2. Fortran modules are not used (or usable) at runtime, only at buildtime of a dependent project
3. Fortran modules depend on using the same compiler for the dependent project as the one that produced the module.
4. As several recipes in conda-forge use modules, we should reflect this in our metadata, rather than let them fall into this trap.
5. To avoid the constraint at runtime, it makes sense to put fortran modules for `foo` into a `foo-devel` package.
6. We should attach constraints to such `-devel` packages to enforce the right compiler.
7. Recipe authors should opt into such a constraint (because we have no way to detect it automatically).
8. This constraint should be on _activated_ compilers, of which we generally have only one per language in 99+% of recipes.

This should hopefully be uncontroversial.

## How to fix this

The obvious first choice would be something that produces a run-export for packages being built with a certain compiler, say something like
```yaml
outputs:
- name: _produces_fortran_modules
build:
string: {{ compiler_fortran }}_{{ PKG_BUILDNUM }}
run_exports:
- _produces_fortran_modules * {{ compiler_fortran }}*
requirements:
run_constrained:
- gfortran_{{ target_platform }} <0.0a0 # [compiler_fortran != "gfortran"]
- flang_{{ target_platform }} <0.0a0 # [compiler_fortran != "flang"]
- ifx_{{ target_platform }} <0.0a0 # [compiler_fortran != "ifx"]
```
(for a worked out example see https://github.com/conda-forge/staged-recipes/pull/30119)

The recipe author of the `foo-devel` package would have to add this to the host-dependencies of that output, and thus the output would gain the respective constraint. The problem is however, that there's no good way to make this constraint of `foo-devel` (in the `host:` environment) for building dependent package `bar` conflict with the fortran compiler (in the `build:` environment).

We could add a strong run-export to the compiler, but that's going too far -- it would also affect the runtime requirements of `bar`, which have nothing to do with fortran modules anymore, and so shouldn't constrain the compiler. This is the missing `host_exports` [problem](https://github.com/conda/ceps/issues/77).

So given that limitation, I went back to the start and reflected that adding some `_magic_package_with_an_underscore` is pretty poor UX for the recipe authors anyway, and that we could solve the problem comprehensively by creating a new compiler key such that the user would then have
```yaml
outputs:
- name: foo-devel
requirements:
build:
- {{ stdlib("c") }}
- {{ compiler("fortran_modules") }} # different key for selecting the compiler!
host:
- [...]
```
This works around the missing host-exports problem because we now _can_ attach a strong run-export to the compilers that we provide under the `fortran_modules:` key (in contrast to the general-purpose compilers which must remain usable also for non-module usage). And because recipe authors won't need to add a magic package manually anymore, we can choose a better name, i.e. drop `_produces`.

This is pretty much the reasoning that @isuruf and I used in https://github.com/conda-forge/flang-activation-feedstock/issues/14 to figure out this solution, but I wanted to get some feedback on this -- from the overall approach down to the bikeshed of naming these things.

Another reason why making this a full-blown compiler sounds like a good idea to me is that our C++ toolchain will have exactly the same problem to solve once C++20 module usage picks up (and it's [starting](https://github.com/conda-forge/libcxx-feedstock/issues/225) to), and therefore we should have a uniform approach across languages. This is also why I prefer the `_modules` suffix for clarity, although Isuru suggested `fortran_mod` (presumably because the fortran modules actually use a `.mod` extension).

In more detail, I would imagine to add the following pins to the global pinning (compiler suffix TBD, but it makes sense to me that this matches whatever we add to the pinning key):
```yaml
cxx_modules_compiler:
- gxx_modules # [linux]
- clangxx_modules # [osx]
- vc_modules # [win] -- modules support in vs2019 is poor; c.f. also #2138
cxx_modules_compiler_version:
# matching cxx_compiler_version
fortran_modules_compiler:
- gfortran_modules # [unix]
- flang_modules # [win]
fortran_modules_compiler_version:
# matching fortran_compiler_version
```

The compiler activation feedstocks (here demonstrated for gfortran) would then gain an extra output along the lines of
```yaml
outputs:
[...]
- name: gfortran_modules_{{ cross_target_platform }}:
build:
run_exports:
strong:
- _fortran_modules * gxx*
requirements:
run:
- {{ pin_subpackage("gfortran_" ~ cross_target_platform, exact=True) }}
```

CC @conda-forge/core

PS. Haha, got a nice issue number! Hopefully by [2525](https://www.youtube.com/watch?v=izQB2-Kmiic), modules will be working smoothly 😆

version-dependence of modules

It's even possible for modules to depend on the compiler version, c.f. the gfortran 15 [release notes](https://gcc.gnu.org/gcc-15/changes.html#fortran):
> The Fortran module `*.mod` format generated by GCC 15 is incompatible with the module format generated by GCC 8 - 14, but GCC 15 can for compatibility still read GCC 8 - 14 created module files.

The approach taken with `_fortran_modules` and `_modules_` is flexible enough to encode some version constraints.

## Maybe `host_exports` is necessary to solve this

_(Edit: updated after some discussion [below](https://github.com/conda-forge/conda-forge.github.io/issues/2525#issuecomment-2903759611)...)_

Actually, the exercise with thinking through the version constraints exposed a potentially fatal flaw in my current proposal.

We always come back to:

> The big problem is how the `foo-devel` modules package (where reasonably we'd have to attach the constraint) in `host:` could be made to conflict with something in `build:`.

And the only way to produce a conflict between `build:` and `host:` is a (strong) run-export. But that defeats the purpose, because we wanted to use regular `{{ compiler("fortran") }}` _without_ a strong run-export for builds that are consuming modules. And in that case, the conflict just doesn't trigger...

So now I'm wondering if a proper solution to all this _requires_ `host_exports` (which would provide the right tool, because then even the general-purpose Fortran compiler could host-export `_fortran_modules * *`; i.e. forcing the right kind of modules in `host:` without producing any dependency change in `run:`).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.