[RFC/tracking issue] deprecating -lib32 multilib system

So, right now we have a system in place that works like this:

• When building i686 packages, and i686 specifically and only, libraries are taken and special -32bit packages are also generated for x86_64 in the special multilib/ repository, whether it's just libraries or full package is then controlled via the mode and so on
• In order to ensure that these libraries can be installed together with native x86_64 ones and not conflict, i686 packages are usually configured to use /usr/lib32 and all other packages are configured to use /usr/lib; through hooks, we make sure everything ends up in /usr/lib in the end, except the -lib32 packages which are populated into lib32 (this is also needed in case programs, e.g. mesa, dynamically load library plugins from somewhere else)
• This is pretty clunky and hacky, and only works for i686 on glibc, and only works with x86_64 on glibc
• It is also potentially problematic when migrating off buildbot, as it means the x86_64 and i686 targets cannot be separated
• But mainly it's a problem for other architectures and musl
• It also used to be a problem that the lib32/lib64 symlinks did not exist in unconfigured chroots, so toolchains had to be configured to always use dynamic linkers from lib and that was terrible and ABI breaking, but has since been mitigated and will be fully put into place with gcc10 upgrade; we could fix this because /usr/lib64 is now a symlink to /usr/lib on 64-bit systems, and /usr/lib32 is a symlink to /usr/lib on 32-bit systems, and these symlinks are a part of base-files, always

Proposed new system

Therefore, I am proposing a new system. The goals of this system should be:

• It should not need any special handling in xbps-src
• It should not need any special treatment by xbps itself
• It should not need generation of any special multilib repos/packages
• It should work on any architecture, any libc with a 32-bit counterpart

There are distributions, mainly Debian, that solve this by having a complete multiarch system in place. That involves having support for handling multiple architectures at once in their package manager, and having everything built in a way that /usr/lib is never used directly, but rather libraries use a special prefix that contains the target triplet, so there are never conflicts. While I considered implementing such system in Void, it quickly turned out to be too large of a can of worms, without that much benefit, which would also require a considerable amount of effort put everywhere. So, instead I am proposing what I call multilib prefixes, which is a simplified, generalized system that only handles the common case of running 32-bit processes on 64-bit systems (or alternatively, 64-bit processes on systems running 64-bit kernel with 32-bit userland).

How do multilib prefixes work?

Before explaining how this is going to work, I'm going to describe the steps needed to get there, it will make it easier. The ones marked so are already done.

• First, we should fix our toolchains to use proper dynamic linker paths again
• Then, we should update the toolchains to use lib32 on 32-bit systems and lib64 on 64-bit systems, unconditionally (this is actually not truly important, as the real libdir is handled separately by build systems, but still, for correctness)
• Then, we should fix xbps-src to allow lib64 paths in addition to lib32, by fixing the xbps-src hooks
• Then, we should fix glibc (done) and other related base bits (crosstoolchains and so on) to use lib32 and lib64 instead of lib
• Similarly, all build-style need to be fixed to follow this (use lib${XBPS_TARGET_WORDSIZE}), and all templates that explicitly set lib32 on i686 need to be generalized (e.g. mesa, pulseaudio and so on; these generally load plugins at runtime, so the path is important there; for most programs, it's not, since library paths are not written in executables/shared libs, only names are, and the lookup paths are controlled by dynamic linker)
• We will also need to find every program/library that does dynamic loading like above and the search path is hardcoded and meaningful there; these templates need to have their revision bumped to rebuild, probably after gcc is updated to 10 and so on - we already did this for most known things, others will be done as they are found
• For musl, we will need to ship custom /etc/ld-musl-ARCHITECTURE.path which will override the default dynlinker search paths to use the appropriate lib32 or lib64 ones - musl by default always uses lib
• For glibc, this can be (and already is, though a bit by accident) handled by ld.so.conf.d
• Fix up locales on musl to always use /usr/share/locale
• Then we will need to introduce handling for the prefixes themselves. More about that later.

Now, to get to how it works, and also why it works.

For one, the /usr/lib path we currently use will be preserved. This is because xbps-src is already updated to intercept anything installing into lib32 or lib64 and properly migrating it to lib using symlinks. This is important.

On 64-bit systems, base-files contains symlinks so that e.g. /usr/lib64 points to /usr/lib. Similar thing happens for lib32 on 32-bit systems. However, since the build systems and so on will automatically default to lib32 or lib64, all library lookups will happen through those symlinks, and not through the actual path. What does this mean?

In order to deal with the case of 32-bit binaries on 64-bit systems, all we have to do is simply create a new Void "root". Let's say this root will be /usr/i686-linux-gnu. This root will contain all the usual files. So your 32-bit libraries will be located in /usr/i686-linux-gnu/usr/lib.

How to install things into this prefix? That is easy; you can just use xbps-install. For example, XBPS_ARCH=i686 xbps-install -r /usr/i686-linux-gnu .... That means you will be able to make use of the entire standard 32-bit repo without any special handling. This is in fact how xbps-src already manages makedepends when cross-compiling.

What is needed now is to hook this prefix into your system. Since 64-bit systems will use lib64 and 32-bit ones will use lib32 for lookups, they are non-conflicting. Therefore, what you can do is make your /usr/lib32 a symlink to /usr/i686-linux-gnu/usr/lib. And that's it; if you run any 32-bit process in your 64-bit system, it will look up its libraries through /usr/lib32, which will point to your custom prefix, containing all the dependencies.

If you think about it, that's not very different from how the current 32bit libs are handled. It's just generalized, and does not use any special tooling.

Of course, this is actually not going to work as is, out of box. There is also the dynamic linker to take care of. This dynamic linker does not always respect lib32 paths or whatever, so it's going to be slightly more involved. Notably on musl, dynamic linker always comes from /lib. E.g. /lib/ld-musl-i686.so.1.

However, the dynamic linkers are the one thing where we are sure there will never be file conflicts. Therefore, you can just symlink the dynamic linker from your multilib prefix to the path where it's expected to be, and everything will just work.

And that is basically the gist of it; there is nothing much more complicated about it conceptually. Of course, there are issues we need to work out, because in practice it's not as smooth.

Problems

• If we choose /usr/<target-triplet> as the default multilib prefix, it will conflict with cross-toolchains. What we could do is integrate this with managing of cross-toolchains on your host system and make them a lot more useful. However, cross-toolchains also provide the libc and other packages that would otherwise be installed. In xbps-src, cross-vpkg-dummy is installed.
• We will need tooling for managing these multilib prefixes. Using just xbps as it is is fairly clunky, because by default these prefixes don't have repos set up in them and so on. Especially initial setup is relatively involved. Another problem is making sure the symlinks are properly set up, for lib32 (or lib64) as well as the dynamic linker symlinks if necessary. The hypothetical management tool could deal with this automatically, besides providing functionality for installing the packages into the prefix.
• Right now, /usr/lib32 is actually not a symlink on 64-bit systems, but a directory. This is because /usr/lib32/locale is a symlink, to your native locale data. We would need to replace that. This is actually a glibc-only problem; application locale data go in /usr/share/locale but glibc puts some autogenerated files in /usr/lib/locale which need to be shared.
• There might be packages that require wordsize-specific data files in /usr/share. These need to be found, and such files need to go to /usr/lib(32|64). This is the right thing to do either way, since they're no different from binary files, /usr/share should only contain architecture-independent data.
• There might be templates where it would be useful to split the libraries manually into a subpackage (foo-libs, for example). This is not a hard requirement, but since multilib prefixes should generally only contain libraries and data files are useless, it might reduce how much disk space we waste.
• There are some "full" 32-bit packages which also contain executable programs. Those would need to be handled specially. Perhaps the hypothetical "prefix management tool" could also take care of installing appropriate symlinks in /usr/bin, probably upon user request.

There might be more issues to work out, these are just ones I can think of right now. I think all of them are solvable, so we should be able to make a concept like this into reality sometime in near future, and deprecate/remove the current multilib repos.

@void-linux/pkg-committers

The issue names xbps-src hooks, build-style, base-files, musl, and prefix management as the remaining areas; start by reviewing the unchecked musl dynamic-linker-path and prefix tasks. Done would require an agreed, working replacement for the -lib32 multilib system, including prefix setup across the targeted architectures and libc implementations.