flutter / flutter/ai

multimodal audio in vs. transcribing an audio attachment?

Open
#43 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Dart
Stars
286
Forks
94
PR merge metrics
No merged PRs in 30d

Description

Currently, it appears that audio is handled as a binary attachment, and is then transcribed. ref https://github.com/flutter/ai/blob/b1fbc7c8dfc6b1b0dc7c2485b4ea2f3d99befa32/lib/src/views/llm_chat_view/llm_chat_view.dart#L230

For multimodal models such as Gemini, audio as an input is natively supported.

The expectation is that instead of an audio attachment that is transcribed, the audio should be used as the input to the model directly rather than the transcription.

Contributor guide

Open the contributing guide

Research direction

Start with lib/src/views/llm_chat_view/llm_chat_view.dart around line 230 and trace how an audio attachment becomes a transcription. Compare that flow with the multimodal model input path. Done means supported audio is passed directly to the model instead of being transcribed first.

Written by the indexing model from the issue text.

Assessment

Tech stack
dart
Domain
ai
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.