Lightning-AI / Lightning-AI/lightning-thunder

[Flaky test] liger_kernel.ipynb

Open
#1,929 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ci / tests
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

Looks like the tolerance on this assertion needs to be widened:

See https://dev.azure.com/Lightning-AI/49263a0c-0f2a-4397-bf3a-60a04485965e/_apis/build/builds/229168/logs/16, relevant part:

```
hidden_size = 64

example_input = torch.randn(32, 10, hidden_size, device=device, requires_grad=True)

with device:
model = litgpt.model.RMSNorm(hidden_size)
thunder_model = thunder.jit(model, executors=[liger_ex])
ref = model(example_input.clone())
res = thunder_model(example_input.clone())
go = torch.randn_like(ref)
grad_ref, grad_ref_weight = torch.autograd.grad(ref, (example_input, model.weight), go)
grad_res, grad_res_weight = torch.autograd.grad(res, (example_input, model.weight), go)

assert liger_rms_norm_forward in {bsym.sym for bsym in thunder.last_traces(thunder_model)[-1].bound_symbols}
assert liger_rms_norm_backward in {bsym.sym for bsym in thunder.last_backward_traces(thunder_model)[-1].bound_symbols}

assert_close(ref, res)
assert_close(grad_ref, grad_res)
assert_close(grad_ref_weight, grad_res_weight)

---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[6], line 20
18 assert_close(ref, res)
19 assert_close(grad_ref, grad_res)
---> 20 assert_close(grad_ref_weight, grad_res_weight)

File /usr/local/lib/python3.10/dist-packages/torch/testing/_comparison.py:1519, in assert_close(actual, expected, allow_subclasses, rtol, atol, equal_nan, check_device, check_dtype, check_layout, check_stride, msg)
1497 error_metas = not_close_error_metas(
1498 actual,
1499 expected,
(...)
1514 msg=msg,
1515 )
1517 if error_metas:
1518 # TODO: compose all metas into one AssertionError
-> 1519 raise error_metas[0].to_error(msg)

AssertionError: Tensor-likes are not close!

Mismatched elements: 1 / 64 (1.6%)
Greatest absolute difference: 1.049041748046875e-05 at index (62,) (up to 1e-05 allowed)
Greatest relative difference: 0.00017781513452064246 at index (62,) (up to 1.3e-06 allowed)
```

cc @borda

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Open liger_kernel.ipynb and inspect the failing grad_ref_weight versus grad_res assertion in the shown cell. Reproduce the flaky failure, adjust the assertion tolerance as needed, and rerun the notebook to confirm the forward and gradient checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python, pytorch
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.