deven96 / deven96/ahnlich

Support Audio Datatypes

Open
#178 0 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Rust
Stars
244
Forks
29
Avg merge
2d 41m
Merged PRs (30d)
19

Description

This task is not fully formed yet, but the idea is to take one step towards supporting audio data.

In the case of text data, tokenization is used to split it into tokens and pass it into the model.
In the case of image data, we resize, convert to ndarray, etc., then pass it into the model.

In the case of audio data???

That is the question this issue looks to answer.

The user should be able to pass in bytes of an audio file and we read and do basic processing (it is fine if this is not model-specific, just generic stuff) that leads to the input type supported by the model.

Such that we can add an extra input type here: https://github.com/deven96/ahnlich/blob/4ed8654b40b1a0930dc1ff7f9aa78bb4f5e0847f/ahnlich/ai/src/engine/ai/models.rs#L247

Such as:

```
#[derive(Debug)]
pub enum ModelInput {
Texts(Vec),
Images(Array),
Audios(...)
}
```

Some other places where this new data type would reflect could be: https://github.com/deven96/ahnlich/blob/4ed8654b40b1a0930dc1ff7f9aa78bb4f5e0847f/ahnlich/ai/src/engine/ai/models.rs#L25-L33

Other useful pointers include:

How we currently read bytes for images: https://github.com/deven96/ahnlich/blob/4ed8654b40b1a0930dc1ff7f9aa78bb4f5e0847f/ahnlich/ai/src/engine/ai/models.rs#L261

The information provided doesn't cover all the bits of code that need modification to make the introduction, so feel free to do what is needed.

Out of scope: Model Support, Specific Preprocessing

Focus should be on being able to get audio into the format that can be sent into models using the most minimal processing possible. For example, in the case of image processing, this would mean simply resizing the image and converting to NdArray.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.