LAION-AI / LAION-AI/CLAP

error with querying audio embeddings using API

Open
#130 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.3k
Forks
213
PR merge metrics
No merged PRs in 30d

Description

Summary
I am trying to extract audio features for the Clotho dataset evaluation examples. However, when I try model.get_audio_embedding_from_filelist and model.get_audio_embedding_from_data I get two different errors.

Code For Reproducing

Downloaded Clotho dataset from https://zenodo.org/record/3490684

import laion_clap
from glob import glob 
import ipdb 
from tqdm import tqdm 
import librosa

model = laion_clap.CLAP_Module(enable_fusion=False)
model.load_ckpt()

print(model)


audio_embeddings = []  
clotho_eval_path = '/clotho/development/'
audio_files = glob(f'{clotho_eval_path}*.wav')

for audio_file in tqdm(audio_files):
    try: 
        audio_data, _ = librosa.load(audio_file, sr=48000) # sample rate should be 48000
        audio_data = audio_data.reshape(1, -1) #Make it (1,T) or (N,T)
        audio_embed = model.get_audio_embedding_from_data(x = audio_data, use_tensor=False) # ERROR
    except Exception as e: 
        print("ERROR1:", e)
    try: 
        audio_embed = model.get_audio_embedding_from_filelist(x = audio_file, use_tensor=False) #ALSO ERRROR
    except Exception as e: 
        print("ERROR2:", e)

    audio_embeddings.append(audio_embed)

And the errors I am getting:

ERROR1: GET was unable to find an engine to execute this computation
ERROR2: /anaconda3/envs/MPD2/lib/python3.8/site-packages/laion_clap/hook.py:137: UserWarning: PySoundFile failed. Trying audioread instead.
  audio_waveform, _ = librosa.load(f, sr=48000)
/anaconda3/envs/MPD2/lib/python3.8/site-packages/librosa/core/audio.py:184: FutureWarning: librosa.core.audio.__audioread_load
        Deprecated as of librosa version 0.10.0.
        It will be removed in librosa version 1.0.
  y, sr_native = __audioread_load(path, offset, duration, dtype)
IsADirectoryError: [Errno 21] Is a directory: '/'

Requirement Versions
librosa 0.10.0
torchlibrosa 0.1.0
transformers 4.30.0

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Clotho reproduction and inspect the get_audio_embedding_from_data and get_audio_embedding_from_filelist entry points. Compare their expected inputs with the reshaped librosa output and audio_file value; done means the documented audio inputs produce embeddings without either reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.