Smoother C/Rust interop with existing libraries.

The issue has no named implementation files, tests, or entry point. Start by reviewing the referenced sys-crate issues and the Rust Embedded Working Group context, then define a narrower, actionable scope from the listed build, linking, and cross-compilation concerns. Done would require an agreed specification or scoped tooling change, but the issue does not currently identify which one.

After attempting to use a collection of c and *-sys libraries in interesting no_std and cross-compiling contexts, it appears to me we have plenty of opportunity to, improve, the state of building and linking existing libraries with rust (and has cost me weeks of my life so far :-/).

While this may not technically be our area of responsibility, I'm not sure where else to put it and I believe it is something that effects us disproportionately (and windows users, trying to compile linked system libraries for windows feels like travelling back in time). If we don't think this should be here (or someone has a better / similar issue) I am happy to close or move the issue elsewhere, but, I think it useful for us to start somewhere ^_^

Also to be clear this is not a critique of existing tooling or the work around this, we have a collection of amazing utilities for use in build.rs and a great ecosystem of crates and bindings. It appears to me that we have a lack of process and abstraction to provide the smooth, packaged, user experience that one hopes for with modern package management, and this is translated to poor publisher experience as it's super difficult to get right, and poor consumer experience as there is no consistency and tooling reliably breaks in all sorts of interesting ways.

Goals

To simplify the construction of *-sys crates that adequately support multiple target architectures, minimising the burden on crate authors to create and manage complex build.rs configurations and improving the crate consumer experience by providing consistent support for a set of required features.

Issues / Requirements / Solutions

I believe most of the issues I have come across could be mitigated through a combination of:

a) the provision of a higher-level c-library integration crate for use in build.rs that abstracts the nuance of building and linking where possible, exposing an opinionated configuration object that manages underlying tools and a set of features to be re-exported by *-sys crates.
b) a spec / guideline for designing *-sys crates with best practices etc.

The following are issues that have arisin during my use of *-sys crates, and some possible mitigations.

*-sys` crates may depend on environmental variables for compilation

This requires a collection of strange environmental variables to be set, on a per-target basis, and is thus difficult to debug or rationalise about for library consumers.

Possible Mitigations

• guide should specify no environmental variables should be required for compilation
• tooling should provide support for per-target-environmental-vars specified in build.rs (and possibly propagated / override from [metadata.PACKAGE][1]

*-sys crates do not build on all (or often, any) no_std or other architectures

This really requires test-builds for all supported architectures

Possible Mitigations

• cross-architecture / target template for building and testing?
• tooling that runs a set of cross-architecture builds?

Pre-built bindgen outputs often do not match target architecture sizes [2] [3]

This results in segfaults and all sorts of other terrible things when running libraries on unexpected architectures

Possible Mitigations

• tooling could build bindgen at compile time, with appropriate target arguments
  • guide should specify that bindgen should be run at compile time
• bindgen could pre-generate a set of variants for different platforms
  • this subset would need to be defined, tested
• packages could default to using pre-packaged bindings, while supporting overrides where required
  • how do we tell whether a binding is valid for a particular architecture in a way that doesn't end in segfaults? do we just need 32-bit and 64-bit versions?

This is likely due to architecture-specific switches in header files, and as such may be too project-dependent for us to provide a good solution :-/

libc does not define types for non tier-1 targets

bindgen defaults to using libc for ffi types, and core::ffi does not contain a number of required types. This means that build scripts often require a modification to use the cty. This often requires patching [4] existing *-sys libraries to detect and alter this behaviour.

Possible Mitigations

• Work is underway to move basic ctypes to the libc repo so the definitions are not empty on unsupported targets [5]
• This may be mitigated by compile-time-bindgen as the correct libc or cty could be selected by target

Package discovery, compilation (and linking) typically doesn't work cross platform

This is a big one, but it's a bit too interrelated to easily split.

*-sys libraries will typically use one mechanism for library discovery, complilation, (and linking), further work is sometimes done to support alternate mechanisms for compilation from source, or static vs. dynamic linking, however, this is often not available and places a huge burden on maintainers to support all the possible variants for a given library.
As an example, using pkg-config which is generally good for unixen tends to not work for windows or static linking (and only sometimes works for cross-compilation under multiarch).

It's also important to be able to link *-sys crates to distro packages for distro packaging, so this probably needs to be specifiable?

Possible Mitigations

• Tooling should be generic over mechanisms for:
  • library / source discovery (pkg-config, vcpkg, local source, fetch from git, curl)
  • compilation (cc, cmake, autoconf etc.)
  • includes (exporting header paths etc.)
  • linking (static, dynamic)

Ideally this would provide a sensible default for your platform, while supporting feature-based overrides as required for different mechanisms for discovery or static compilation etc, as well as configuration overrides from cargo metadata. This way it's reasonably simple to configure per-target build options if required, and to fix build.rs paths etc. from the top level package.

newlib requires function stubs

Many libraries that support embedded compilation via autoconf or other complex mechanism both depend on the standard c libraries and sometimes make a bunch of decisions about linking things like newlib, which then requires function stubs in the rust application.

Possible mitigations

• Support specifying these linker args as part of standard tooling (or, provide improved documentation on the use of .cargo/config for this purpose) (relates to environmental args issue)
• Provide a newlib_stubs and/or libc_stubs that exports a set of c compatible stubs that use the underlying rust allocator etc.

---

If you have come across other issues / solutions, or demonstration of these issues, or have any other thoughts / opinions, please post and I will update the list here ^_^

Other References

• https://users.rust-lang.org/t/improving-the-sys-crate-story/46097
• https://kornel.ski/rust-sys-crate

Existing *-sys library issues

• https://github.com/sodiumoxide/sodiumoxide/issues/377
• https://github.com/dcuddeback/libusb-rs/issues/20