Provide Attention scores from Transformer
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python, pytorch
- Domain
- machine-learning
Research direction
Start in torchaudio/models/wav2vec2/components.py around line 317, where the attention scores are available, and inspect the return at line 326. Determine how the Transformer output should expose those scores alongside the representations. Done means callers can receive the attention scores rather than always getting None.
Written by the indexing model from the issue text.
Description
🚀 The feature
Thanks for your amazing contributions.
As far as I understand, the Transformer encoder employed in torchaudio does not provide attention scores in their outputs. Otherwise, please ignore this thread and let me know.
The following line can be saved as attention scores, then can be provided in line 326 return.
Instead, the current implementation does not return but only the representations of vectors. Line 326: return output, None
Motivation, pitch
The attention scores of Transformer encoder are very valuable information to design more advanced models. Huggingface implementation allows it by configurations, and it allows other AI researchers to explore new studies, such as the model predictions considering attention scores, the loss function considering attention scores as well.
Alternatives
No response
Additional context
No response
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 799
- Avg merge
- 58m
- Merged PRs (30d)
- 3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from pytorch/audio
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
-
Difficulty 1/5 Under an hour Newbie friendliness 65/100
-
Difficulty 3/5 1-2 days Newbie friendliness 72/100
-
transforms.Vad silently returns an empty tensor when the waveform contains one non-finite sample Open
Difficulty 3/5 1-2 days Newbie friendliness 70/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100