tensorflow / tensorflow/text

not accurate in the transformer implementation comment

Open
#1,132 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.3k
Forks
379
Avg merge
3h 30m
Merged PRs (30d)
8

Description

I read about the tutorial of using Transformer to translate the pt to en in tutorial. In the data pipeline construction part :

MAX_TOKENS=128
def prepare_batch(pt, en):
    pt = tokenizers.pt.tokenize(pt)      # Output is ragged.
    pt = pt[:, :MAX_TOKENS]    # Trim to MAX_TOKENS.
    pt = pt.to_tensor()  # Convert to 0-padded dense Tensor

    en = tokenizers.en.tokenize(en)
    en = en[:, :(MAX_TOKENS+1)]
    en_inputs = en[:, :-1].to_tensor()  # Drop the [END] tokens
    en_labels = en[:, 1:].to_tensor()   # Drop the [START] tokens

    return (pt, en_inputs), en_labels

the comment which is attached to the line en_inputs = en[:, :-1].to_tensor() is not precise and may cause confusion among beginners. If the english sentence has more than MAX_TOKENS tokens, then the last token of the en is not [END] (it has been sliced). So in this situation, it's not right to say en_inputs = en[:, :-1].to_tensor() is used to drop the [END] tokens. I am stuck in this problem when I am dealing with my own training data. The actual case is whatever the length the sequence has, we only take the [MAX_TOKENS] tokens. It means that in some cases, the target sequence which we input to the decoder may not have [END] in the end of the sequence.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TensorFlow Text Transformer tutorial linked in the issue and inspect its data pipeline section around prepare_batch. Verify how slicing affects the final token when sequences exceed MAX_TOKENS, then update the en_inputs comment to describe the behavior accurately. Done means the tutorial no longer implies that an [END] token is always removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.