How to return_attention from Generator.generate_tokens ?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 536
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
Hi,
I would like to access attention from decoder-only LLMs. I need it for simultaneous translation, to detect in each generation step, to which part of source (user message in the prompt) the model is attending most, to see if it is near the end of current source. Then I would stop generation, to continue with next partial source.
Am I right that this feature is not yet implemented for Generator, but is there for Translator? Could it be implemented for Generator? How?
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the existing Translator attention behavior with the Generator.generate_tokens entry point and determine whether attention is currently exposed there. Clarify the API and generation-step data needed to return decoder attention for simultaneous translation; done means Generator can provide the requested attention information, with matching documentation or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100