google / google/flax

nnx.make_causal_mask() usage

Open
#4,505 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
7.3k
Forks
833
Avg merge
5h 11m
Merged PRs (30d)
5

Description

So this is a follow-up on #4290 (@cgarciae). For building a causal LM, I need to use causal masking. Here is my attempt (by adding a single line using the code from #4290:

```
batch_size = 2
seqlen = 40
emb_size = 256

x = jnp.ones((batch_size, seqlen, emb_size))

mha = nnx.MultiHeadAttention(
in_features=emb_size, num_heads=2, decode=True, rngs=nnx.Rngs(0)
)
shape = x.shape

for i in range(seqlen): # iterate all tokens
y = mha(inputs_q=x[:, i : i + 1],
mask=nnx.make_causal_mask(x[:, i : i + 1])) #newly added
```

The error I got is:

```
AssertionError: masks must have same rank: (5, 4)
```

I cannot make sense of this error :(

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.