New Plugin API
- Dominant language
- C++
- Stars
- 10.4k
- Forks
- 1.3k
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 7
Description
### Enhancement Summary
Over the past couple years, I've mentioned in various discussion threads how I want to rewrite the LMMS plugin API and the way plugins and LMMS interact more broadly. However, I've never really collected these ideas together in one place, so that's what I'm doing here.
## Motivation
There are *many* reasons why we should rework the plugin system
- To support 3rd party native LMMS plugins (currently all plugins must exist in-tree and be built alongside the main application)
- To implement CLAP support without breaking the CLAP API's [threading rules](https://github.com/free-audio/clap/blob/29ffcc273be7c7c651f6c9953b99e69700e2387a/include/clap/ext/thread-check.h#L13-L55)
- To implement CLAP and VST3 support without the need for compromises or hacks, following established conventions for how all other major audio plugin APIs work
- To improve thread safety in general
- To improve real-time safety
- To eventually implement a robust remote plugin system where any plugin can be launched in either a remote process or in the LMMS application's own process without the plugin's awareness
- To lessen our reliance on Qt for plugins
- To make the plugin system more easily extendable in the future
- To clean up the cruft that has accummulated over the years
- ...
## Design
I have some familiarity with the VST2/3 and LV2 APIs, though I'm most familiar with CLAP and have made some minor contributions to the CLAP API myself. While I may be biased, CLAP is a very clean, well-specified, extensible, and easy-to-use API which makes it appealing.
So for the new LMMS plugin API, I favor a design that takes heavy inspiration from CLAP - basically a C++ version of parts of the CLAP API with modifications for integrating with our views, models, and other components.
The design should:
- Maintain a clear separation of plugin and host responsibilities
- Introduce plugin states (active/inactive, processing/not-processing, etc.)
- Specify which actions are allowed during which states
- Specify which methods are allowed to be called on which threads
- Verify correct state transitions and compliance with threading rules - at least in debug builds
- Rework how events are passed between the plugin and host so they are both thread-safe and real-time safe
- Be easily adaptable to all major 3rd party plugin APIs (CLAP, LV2, VST3, etc.) as well as our own native plugins
- Be easily extensible when plugins need access to more host capabilities/info, or vice versa
- Coordinate design with a future plugin manager
- Coordinate design with a future `RemotePlugin` rewrite
- Coordinate design with a future preset database
- ...
## Implementation
Here are my plans as they currently stand.
- Plugin/Host separation
- No more implicit dynamic linking of plugins to LMMS [as seen here](https://github.com/LMMS/lmms/blob/a8423c1e4a25f67fd350588bef7704a3bbd003ff/cmake/modules/BuildPlugin.cmake#L54)
- No more Wild West where any random class from the LMMS application can be given `LMMS_EXPORT` and directly used from a plugin
- Forces the use of a well-defined plugin API for ALL communication between plugins and host
- Allows for 3rd party LMMS plugins that do not live in-tree
- Remove plugin dependence on Qt as much as possible
- Support multiple plugin types in one binary
- Currently LV2 and VST implementations require duplicate Instrument and Effect versions that live in separate dynamic shared objects (DSOs - the DLL/so/dylib).
Instead, the plugin API should support multiple plugin types coexisting in the same DSO.
- LV2 and VSTs do not need to be refactored to take advantage of this feature right away but they could in the future with an upgrade routine. CLAP and VST3 could work this way from the start.
- `include/lmms` public header directory
- `include/lmms/plugin` directory for all public plugin headers
- `include/lmms/README.md` contains information for developers getting started with LMMS plugin development
- `include/lmms/plugin/clap/lmms-ext.h` if we ever offer a CLAP API extension for LMMS
- `lmms::plugin` CMake target for linking with these headers
- A header called `Plugin.h` could include all plugin headers for convenience
Usage would look like:
```cpp
#include
```
- Introduce an `lmms::plugin` namespace? (Unsure about this)
- Define two plugin threads - main and audio - and a way to check the current thread
- Plugin states
- Plugins (whether instrument or effect) need to have a well-defined set of states they can exist in, with state transitions also clearly defined
- Correct state transitions should be verified through `assert`s or other means
- Plugins will follow the states and state transitions CLAP uses ([link](https://github.com/free-audio/clap/blob/main/include/clap/plugin.h))
- States are a key part of ensuring correct operation of the plugin
- Currently, LMMS plugins really only have instantiation and destruction, and everything that happens in between is completely unspecified. Plugins have to use mutexes to protect the process loop during certain user interactions on the main thread such as changing the preset, loading a new VST, opening the ZynAddSubFX GUI, or even just playing notes, but this would not be needed with a well-designed API.
- Add an extension system
- This would remove the need for adding a bunch of random virtual methods to Instrument/Effect/Plugin classes which may or may not be overridden by plugin implementations
- The host side could also use the extension system to provide plugins with access to any random info or host capabilities a plugin may need: transport info, microtuner settings, etc.
- Add a `PluginId` class for uniquely identifying a plugin
- Replaces the incomprehensible `Plugin::Descriptor::SubPluginFeatures::Key` system
- Works the same way for both regular plugins and "sub-plugins"
- Provides conversions to/from a unique ID string
- Add a `PluginDescriptor` class
- Replaces the `Plugin::Descriptor` and `Plugin::Descriptor::SubPluginFeatures` system
- Works for both regular plugins and sub-plugins without any weird special-casing
- Add a `Host` or `HostInstance` class (name TBD) for plugins to interact with the host
- The host passes an instance of this class to a plugin when it is instantiated
- Each plugin instance receives a unique `Host` object, so when the plugin uses methods in it, the host knows which plugin is calling
- The `Host` class provides:
- the LMMS version
- the LMMS plugin API version
- Access to host extensions
- the plugin's `Model`
- methods for requesting changes to the plugin's state (see below)
- Add a `PluginEntry` class
- Plugins implement this class and expose it to the host via an exported `lmms_plugin_entry` symbol
- Replaces current `*_plugin_descriptor` and `lmms_plugin_main` symbols
- Supports plugin DSOs with multiple sub-plugins
- Provides the `PluginDescriptor` plugin descriptor(s)
- Provides init/deinit methods for the plugin's DSO
- Provides information about "sub-plugins"
- For LV2, CLAP, etc. the "sub-plugins" need to be discovered, which can be time consuming, so there should be a means of discovering plugins in a background thread
- Provides method for instantiating a plugin, given its `PluginId` and a `Host*` pointer
- Eventually should also provide a preset discovery mechanism which can also run in a background thread
- Add an event queue for MIDI events and other events
- Lock-free and real-time safe
- Replaces the `handleMidiEvent` method
- Two queues for bidirectional communication (host-->plugin + plugin-->host)
- The host should maintain some data about each plugin instance beyond just its `Plugin*` pointer
- This data would host-only and invisible to plugins (not in the public plugin headers)
- Could be called `PluginInstance` maybe
- Data could include the pin connector's `AudioPortsModel`, the plugin's buffers, and play handles (?)
- Rework the process method for Instruments (see [#8294](https://github.com/LMMS/lmms/issues/8294) for more info)
- Multi-stream instruments are `NotePlayHandle`-based instruments which have an audio buffer associated with each playing note. This allows LMMS to apply volume envelopes to each note individually. In CLAP, this would require changing the number of audio ports dynamically with each NoteOn and NoteOff event, which is not supported. So we'd either need a CLAP extension to support dynamically adding/removing buffers and associating them with notes (voices), or we'd need a new, better design that does the same thing. Such as an ADSR extension which tells the plugin how to perform ADSR, and after the plugin does so, it mixes the results into a single buffer - eliminating the need for a dynamic number of buffers passed between plugin and host.
- ...
I have been working on the `PluginId`, `PluginDescriptor`, and `PluginEntry` classes locally, but I do not have a PR ready yet.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.