Asset System Rewrite
- Dominant language
- C++
- Stars
- 10.4k
- Forks
- 1.3k
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 7
Description
# Enhancement Summary
For all kinds of data, from projects to plugins to samples, LMMS currently manages most of its saving and loading of that data through the `DataFile` class. `DataFile` does not offer its own built-in serialization methods, but rather relies on individual classes to write to and read from a separate `DataFile` that then gets serialized as XML.
In short, the `DataFile` workflow was something that grew, rather than being engineered. From what I gather, most parts of `DataFile` are written in a way that is inflexible and outdated; I don't think some of that code has seen the light of day in years :smiling_face_with_tear:. LMMS as a whole would benefit from a more modular, extensible, and flexible asset handling system to replace it.
# Justification
This will allow us to do several things much more easily:
- Add support for saving and loading new file types like audio file formats, plugin settings, etc. Most of `DataFile` is hard-coded with enums and fixed lists of file extensions, and the code does not lend itself well to changes to those. A system where the asset importer is decoupled from the asset types themselves would make it far easier to do this.
- Remove Qt from core. DataFile is heavily based on QDomDocument, which I don't think was ever designed to be used for the way `DataFile` uses it, and removing it would be beneficial to the goal of #6472.
- More easily change the serialization method backing project files. Currently, we use XML, but there has also been interest expressed in a JSON format (#3981), which would be nigh on impossible to accomplish without a total rewrite of DataFile anyways. More on this point will come later.
- Fix bugs and write unit tests. Broadly speaking, I think modernized, better engineered code is always easier to debug and fix problems in, but in this specific case, I do think that specifically, the removal of Qt and decoupling of individual parts would make life better. #1191 is an example of a bug that would be much easier to fix with a new system. As I've been saying ad nauseam, the `DataFile` API is pretty convoluted, and when I tried to write tests for it in #7121, I found it was not easy to try and write comprehensive tests, which makes it hard to guard the system against regressions.
# Mockup
I've been thinking about how the new system would need to be designed to address the above points, and this is what I've come up with:
`Asset`
---------------------------------
All types that can be considered "project data objects" will inherit from the `Asset` base class. The `Asset` class will have away of indicating what type of asset it is, and a specific way of binding properties that are to be serialized, with an API something like this:
```cpp
// The name of this type of asset. Replaces the DataFile type enum with something more flexible.
// Might return "Sample" or "MIDI"
std::string assetType() const;
static std::string assetType() const;
std::vector getPropertyList() const; // return names of exposed members.
template
T getProperty(std::string property_name) const;
template
void setProperty(T &new_value);
```
Basically, it would be similar in concept to Blender's RNA system, or Godot's `Object` class.
`AssetLoader`
--------------------------------------------------
Asset loading will be handled by the `AssetLoader` *(maybe `AssetReader`, but I don't like that as much)* abstract class, with an API something like this at minimum:
```cpp
bool handlesFile(std::string file_path) virtual const; // whether or not this asset loader is capable of loading a specific file.
Asset import(std::string file_path) virtual; // imports the file
Asset load(std::string file_path) virtual; // loads an already imported file
```
I see `AssetLoader`s being implemented in a hierarchical, modular way. The base asset loader used by LMMS would be `AssetLoaderDefault`, which would be a wrapper around several `AssetCategoryLoader`s, which load specific file types. There will also be an `AssetLoaderText` base class which loads serialized properties of previously saved assets. This diagram illustrates it best:

`AssetWriter`
------------------------------------------------------------------
Asset saving will be handled by the `AssetWriter` class. There will be an `AssetWriterDefault` which takes the exposed properties and stores them in a structured format (probably text), and will recursively serialize all `Asset` properties of an `Asset`. This could be further specialized into something like an `AssetTextWriterXML` / `AssetTextWriterJSON` class if desired.
Example API:
```cpp
void save(Asset asset, std::string file_path, bool compressed = true) virtual; // Writes an asset to the given location on disk.
```
Here's a diagram broadly illustrating how the saving of a song might work in practice:

Both this and `AssetLoader` would ideally be based on `std::filesystem`.
------------------------------
In conclusion, this would be a major change and represent a significant amount of effort, but I believe it is something that needs to happen if LMMS is to move forward.
I would very much appreciate feedback and debate on the idea, because I am under no illusions that the design I have laid out here is the best possible one, and it could be made even better with input from other, more experienced developers.
Thank you!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the existing DataFile class and the testing attempt referenced in #7121, then trace how projects, plugins, and samples are currently serialized. The issue proposes Asset, AssetLoader, and AssetWriter entry points, but does not define concrete files or acceptance tests; completion would require an agreed design and substantial implementation replacing the current workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend-api-design, desktop
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100