sillsdev / sillsdev/machine.py

Implement ATT-INPUT method for computing alignments from cross-attention

Open
#261 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
12
Forks
3
Avg merge
2d 3h
Merged PRs (30d)
5

Description

The HuggingFaceNmtEngine class currently implements the ATT-OUTPUT approach from this paper. The ATT-INPUT method would generate better quality alignments. In order to implement ATT-INPUT, the class would need to shift the attentions to the left one step. This can be done by not adding a 0 matrix at the beginning of the attentions. We would also need to change what layer the attentions are retrieved from (bottom layers). For ATT-INPUT, it is possible for the last token to not get aligned if the translation has hit the max generation length. This edge case should be handled properly.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the HuggingFaceNmtEngine implementation and compare its current ATT-OUTPUT attention handling with the linked paper. Trace where the leading zero matrix and attention layer selection are applied, then determine how ATT-INPUT should handle translations that reach the maximum generation length without aligning the final token. Done means ATT-INPUT produces the intended alignments, including that edge case.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.