Building cdylibs and plugins with cargo

There's a few open issues related to building plugins for other software (as cdylibs) with cargo. With this issue I want to summarize them and try to work to a single solution.

The use case

Software like Python, PostgreSQL, and many others can dynamically load plugins. These plugins are dynamic libraries (i.e. cdylibs in Rust, aka .so/.dylib/.dll files) with certain symbols exposed. We're looking here at making such plugins using Rust.

The problems

1. It's unclear whether these should be [lib] or [[bin]] crates. [[bin]]s cannot be cdylibs, but a package can currently only contain one [lib]. These plugins will not be 'consumed' by other Rust crates as dependencies, so this limitation is unnecessary for this type of crate. #6351
2. A cdylib [lib] is not made available to integration tests to test. #8311 #7152 #8193
3. required-features = .. is ignored for [lib].
4. The output file name cannot be chosen. E.g. on Linux it will be lib{cratename}.so, even though it might need to be something like {cratename}.plugin, python-{cratename}.pyd, etc. https://github.com/rust-lang/cargo/issues/1706#issuecomment-442891843
5. Plugins might need to depend on symbols from the program they will be loaded in to. This requires -Clink-arg=-undefined=dynamic_lookup on Mac. It might also require not depending on a library that a regular bin target should depend on. #5881
6. It is assumed the [lib] crate will be linked into the [[bin]] crates: dependencies from build.rs are added to the [lib] crate but not the [[bin]] crates, to avoid duplicate symbols. Plugin crates should not be linked into [[bin]] crates. #7506
7. These type of crates should set enable Package::include_lockfile(). This is done for examples and binaries, but not for libraries.
8. Sometimes these files will also have an entry point, so you can run them as binary (like how GNU libc has an entry point that prints its version). Rustc doesn't make an entry point for cdylibs by default, and bin targets don't handle thread locals in a way that they can be loaded into other binaries. (Probably not a cargo issue.)
9. Maybe more?

Possible solutions

#6351 suggests allowing crate-type = ["cdylib"] for [[bin]]s. Although these type of crates share some properties with [[bin]] crates, there's also properties they do not share: Binaries should be installed with cargo install, but plugins should probably not (or at least not into /usr/bin). Binaries have their private items documented by default (as they have no public items), but plugin crates do have public items. Binaries can be run by cargo run, but these should not.

https://github.com/rust-lang/cargo/issues/6351#issuecomment-633682243 suggests adding a new [[clib]] instead. This would be a much bigger change to cargo, as the set of target types (lib, bins, examples, tests, benches) are used in a lot of places (including command line flags).

Questions

1. Is a 'plugin' fundamentally different from a binary? Should they be [[bin]]s, or do they need more special treatment?

2. If not [[bin]]s, are they fundamentally different from other cdylibs? Would a [[clib]] work for both plugins and other type of dynamic C libraries? Or does it need a separate [[cplugin]] or cplugin = true in Cargo.toml to be handled correctly?

3. Should 'cdylibs with an entry point' be supported? If so, as "libraries that also have an entry point" (e.g. [[clib]] also_bin = true), or as "binaries that are also loadable" (e.g. [[bin]] export_stuff = true)?

4. Probably more questions?

Begin with the linked issues #6351, #8311, #7152, #8193, #7506, and #5881, then review Cargo’s existing target categories and the plugin use cases listed here. Done means reaching a settled model for cdylibs and plugins with a defined implementation scope; the payload names no source files or tests.