Proposal for improving native dependencies management

This is an issue mostly not about Vector itself, but about making its native dependencies more portable.

However, initially I want to discuss the proposal here to understand does it solve Vector's problems.

Introduction

Normally, native dependencies are managed in Rust by conventionally using *-sys crates. Each *-sys crate usually contains a build script performing the following steps:

• runs ./configure or cmake with appropriate flags
• runs make
• prints cargo:rustc-link-* strings to let know Rust linker about static libraries produced on the previous step

This works well for native dependencies that do not have dependencies themselfs except standard C and C++ libraries. However, for native libraries that do have dependencies things become more problematic and I'm not aware of a standard way in Rust community to handle this.

Examples:

• leveldb-sys crate with snappy feature depends on Snappy library. Snappy is shipped alongside the code of the crate and is always built in statically.
• rdkafka-sys crate with ssl feature depends on OpenSSL. The crate doesn't ship it, but instead assumes that it is installed on the system.

The main idea

The main idea is to use *-sys crates to provide native dependencies for native dependencies. Each *-sys crate could export not only linker flags, but also include directories (and potentially other relevant information) to build scripts of the crates that use it. Then these exported include directories and linker flags could be used to set configure flags in a correct way.

Such method supports having both native and system-wide versions of native dependencies, including dynamically linked ones. For example, if openssl-sys crate is set to use system-wide SSL, it would export include directories and linker flags for dynamic linking to system-wide SSL. Otherwise, if the vendored feature is enabled, it would export include directories of the vendored OpenSSL and linker flags for statically linking to it.

It would allow to avoid possible duplication and not depend on the system to provide nested native dependencies. For example, if we decide to use Snappy not only in leveldb-sys crate, but also in rdkafka-sys crate to support compressed Kafka topics, currently we would end up with two copies of Snappy in the binary. With this approach there would be only one copy coming from a snappy-sys crate.

The exact mechanism for doing this is yet to be specified.

Problems this approach would solve:

• Reduce minimal build-time dependencies for users who don't need dynamic linking down to only Rust and C/C++ toolchains, without assumption of any additional native libraries (such as OpenSSL, which is an additional dependency now)
• Simplify cross-compilation

Application to Vector

Once we figure out exact way for exporting and using these include/libs information for native dependencies, we could

1. Try to implement it in our forks of openssl-sys, rdkafka-sys, and leveldb-sys.
2. Document this approach more throughly and discuss it with maintainers of these crates and wider Rust community.