pytorch / pytorch/executorch

[Android] Kotlin API improvements for audio LLM models

Open
#19,817 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
581

Description

🚀 The feature, motivation and pitch

ExecuTorch's Android API already has foundational audio support (LlmModule.prefillAudio, AsrModule, C++ MultimodalRunner), but several gaps remain for running decoder-only audio LLMs like Voxtral smoothly from Java/Kotlin.

Current State

LlmModule has prefillAudio(float[]/byte[], ...) and prefillRawAudio(byte[], ...) -- works but limited
AsrModule is encoder-decoder specific (Whisper-style) -- not applicable to decoder-only audio LLMs
Model type constants: only MODEL_TYPE_TEXT (1) and MODEL_TYPE_MULTIMODAL (2) -- no audio-specific semantics

Proposed Improvements

  1. Add ByteBuffer variants for audio prefill (zero-copy parity with image API)

Image prefill has prefillImages(ByteBuffer, ...) for zero-copy JNI, but audio prefill only accepts arrays. For long audio (Voxtral handles multi-minute clips), the extra JNI copy is wasteful.
kotlin
// Missing today:
fun prefillAudio(audio: ByteBuffer, batchSize: Int, nBins: Int, nFrames: Int)
fun prefillRawAudio(audio: ByteBuffer, batchSize: Int, nChannels: Int, nSamples: Int)
2. Add WAV file path API to LlmModule

AsrModule can accept WAV paths directly (internally uses load_wav_audio_data()), but LlmModule requires the caller to manually decode audio in Java. For audio LLMs this should be as easy as:
kotlin
llmModule.prefillAudioFromFile("/path/to/audio.wav")
3. Add MODEL_TYPE_TEXT_AUDIO constant

Currently MODEL_TYPE_MULTIMODAL (aliased as MODEL_TYPE_TEXT_VISION) has vision-centric naming. Adding an explicit audio model type improves discoverability and documentation:
kotlin
const val MODEL_TYPE_TEXT_AUDIO = 3
4. Better raw audio type support

prefillRawAudio(byte[], ...) is awkward -- real PCM audio is typically short[] (16-bit) or float[] (32-bit). Add typed variants:
kotlin
fun prefillRawAudio(audio: ShortArray, batchSize: Int, nChannels: Int, nSamples: Int) // PCM-16
fun prefillRawAudio(audio: FloatArray, batchSize: Int, nChannels: Int, nSamples: Int) // float32
5. Audio-specific configuration in LlmModuleConfig

Add fields for audio preprocessing parameters:
kotlin
data class LlmModuleConfig(
// ... existing fields ...
val sampleRate: Int = 16000,
val preprocessorPath: String? = null, // optional .pte for mel spectrogram extraction
)
6. Unified multimodal generation entry point

Currently audio prefill and text generation are separate calls with no way to combine them in a single config. Consider a builder pattern:
kotlin
llmModule.generate {
audio("/path/to/audio.wav")
prompt("Transcribe the above audio:")
maxSeqLen(512)
onToken { token -> /* stream */ }
}

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the existing LlmModule prefillAudio and prefillRawAudio APIs, the image ByteBuffer API, and AsrModule's WAV handling through load_wav_audio_data(). Map these entry points and LlmModuleConfig before narrowing the six proposals to an agreed scope. Done requires a defined API plan and validation for the selected Android audio-LLM improvements.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, cpp, java, kotlin
Domain
ai, machine-learning, mobile
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.