jina-ai / jina-ai/executors

AudioCLIP eval mode gives different output

Open
#263 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
31
Forks
12
PR merge metrics
No merged PRs in 30d

Description

I was looking at the `AudioCLIPEncoder` and tried running the AudioCLIP demo here:

https://github.com/AndreyGuzhov/AudioCLIP/blob/master/demo/AudioCLIP.ipynb

I found that if we replace this line:

`aclp = AudioCLIP(pretrained=f'../assets/{MODEL_FILENAME}')`

with

`aclp = AudioCLIP(pretrained=f'../assets/{MODEL_FILENAME}').eval()`

the results returned are different and accuracy is lower in eval mode.

Unfortunately, since the original mode does not support input of batch size 1, we are currently using `eval` mode in our `AudioCLIPEncoder` .

I will file an issue in the `AudioCLIP` repo, but also filing an issue here to keep track of things.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.