tensorflow / tensorflow/text

Potential mistake in positional encoding example

Open
#1,122 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.3k
Forks
379
Avg merge
3h 30m
Merged PRs (30d)
8

Description

Hello,
The example notebook for creating a Transformer network features a section on how to add a positional encoding to the input sequence.

As basis, the formula from Attention Is All You Need is given:

$\Large{PE_{(pos, 2i)} = \sin(pos / 10000^{2i / d_{model} })}$
$\Large{PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i / d_{model} })}$

To my understanding, the here described PE does alternate between sine and cosine functions for even and odd feature dimensions.

The implementation in the tutorial however calculates sine functions for the first half of feature dimensions and cosine functions for the second half:

def positional_encoding(length, depth):
    depth = depth/2

    positions = np.arange(length)[:, np.newaxis]     # (seq, 1)
    depths = np.arange(depth)[np.newaxis, :]/depth   # (1, depth)

    angle_rates = 1 / (10000**depths)         # (1, depth)
    angle_rads = positions * angle_rates      # (pos, depth)

    pos_encoding = np.concatenate(
        [np.sin(angle_rads), np.cos(angle_rads)],
        axis=-1
    ) 

    return tf.cast(pos_encoding, dtype=tf.float32)

Am I missing something here or is this implementation different from the mathematical definition?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Open docs/tutorials/transformer.ipynb and inspect the positional_encoding function alongside the displayed Attention Is All You Need formulas. Verify whether the sine/cosine ordering is intentional or inconsistent; done means resolving the reported mismatch in the example or its explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, numpy, python, tensorflow
Domain
documentation, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.