Potential mistake in positional encoding example
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.3k
- Forks
- 379
- Avg merge
- 3h 30m
- Merged PRs (30d)
- 8
Description
Hello,
The example notebook for creating a Transformer network features a section on how to add a positional encoding to the input sequence.
As basis, the formula from Attention Is All You Need is given:
$\Large{PE_{(pos, 2i)} = \sin(pos / 10000^{2i / d_{model} })}$
$\Large{PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i / d_{model} })}$
To my understanding, the here described PE does alternate between sine and cosine functions for even and odd feature dimensions.
The implementation in the tutorial however calculates sine functions for the first half of feature dimensions and cosine functions for the second half:
def positional_encoding(length, depth):
depth = depth/2
positions = np.arange(length)[:, np.newaxis] # (seq, 1)
depths = np.arange(depth)[np.newaxis, :]/depth # (1, depth)
angle_rates = 1 / (10000**depths) # (1, depth)
angle_rads = positions * angle_rates # (pos, depth)
pos_encoding = np.concatenate(
[np.sin(angle_rads), np.cos(angle_rads)],
axis=-1
)
return tf.cast(pos_encoding, dtype=tf.float32)
Am I missing something here or is this implementation different from the mathematical definition?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Open docs/tutorials/transformer.ipynb and inspect the positional_encoding function alongside the displayed Attention Is All You Need formulas. Verify whether the sine/cosine ordering is intentional or inconsistent; done means resolving the reported mismatch in the example or its explanation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, numpy, python, tensorflow
- Domain
- documentation, machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100