LAION-AI / LAION-AI/emotion-annotations

Annotate_audio.py not outputting transcription.

Open
#3 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
112
Forks
11
PR merge metrics
No merged PRs in 30d

Description

Hi,
Thanks for creating this great project.
The annotate_audio.py is working well to give emotion annotations, but does not output an accompanying transcription of the speech. This may be my mis-understanding of your use of the word 'transcription'. Your Example Output suggests caption contains 'transcribed text' but actually is a high level description of the text. It would be worth improving naming conventions or adding transcription of spoken text to avoid further confusion.

Thanks,
Caspar

E.g my_audio_file.json:

    "caption": "AA medium-quality recording of a male speaker describing a painting. The speaker sounds calm and informative, with a slightly nostalgic tone. The recording quality is decent, with no noticeable background noise.",

There are no obvious error messages although in the console. Although perhaps these are relevant.

Transcription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English. This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`. See https://github.com/huggingface/transformers/pull/28687 for more details.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.  

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with annotate_audio.py and the README's Example Output to compare the documented meaning of caption with the generated JSON. Determine whether the project should rename the field or add spoken-text transcription, then verify the chosen behavior against the sample output and the reported Whisper messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
audio-video-rtc, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.