Lightning-AI / Lightning-AI/lightning-thunder
[Flaky test] liger_kernel.ipynb
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
Looks like the tolerance on this assertion needs to be widened:
See https://dev.azure.com/Lightning-AI/49263a0c-0f2a-4397-bf3a-60a04485965e/_apis/build/builds/229168/logs/16, relevant part:
```
hidden_size = 64
example_input = torch.randn(32, 10, hidden_size, device=device, requires_grad=True)
with device:
model = litgpt.model.RMSNorm(hidden_size)
thunder_model = thunder.jit(model, executors=[liger_ex])
ref = model(example_input.clone())
res = thunder_model(example_input.clone())
go = torch.randn_like(ref)
grad_ref, grad_ref_weight = torch.autograd.grad(ref, (example_input, model.weight), go)
grad_res, grad_res_weight = torch.autograd.grad(res, (example_input, model.weight), go)
assert liger_rms_norm_forward in {bsym.sym for bsym in thunder.last_traces(thunder_model)[-1].bound_symbols}
assert liger_rms_norm_backward in {bsym.sym for bsym in thunder.last_backward_traces(thunder_model)[-1].bound_symbols}
assert_close(ref, res)
assert_close(grad_ref, grad_res)
assert_close(grad_ref_weight, grad_res_weight)
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[6], line 20
18 assert_close(ref, res)
19 assert_close(grad_ref, grad_res)
---> 20 assert_close(grad_ref_weight, grad_res_weight)
File /usr/local/lib/python3.10/dist-packages/torch/testing/_comparison.py:1519, in assert_close(actual, expected, allow_subclasses, rtol, atol, equal_nan, check_device, check_dtype, check_layout, check_stride, msg)
1497 error_metas = not_close_error_metas(
1498 actual,
1499 expected,
(...)
1514 msg=msg,
1515 )
1517 if error_metas:
1518 # TODO: compose all metas into one AssertionError
-> 1519 raise error_metas[0].to_error(msg)
AssertionError: Tensor-likes are not close!
Mismatched elements: 1 / 64 (1.6%)
Greatest absolute difference: 1.049041748046875e-05 at index (62,) (up to 1e-05 allowed)
Greatest relative difference: 0.00017781513452064246 at index (62,) (up to 1.3e-06 allowed)
```
cc @borda
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Open liger_kernel.ipynb and inspect the failing grad_ref_weight versus grad_res assertion in the shown cell. Reproduce the flaky failure, adjust the assertion tolerance as needed, and rerun the notebook to confirm the forward and gradient checks pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python, pytorch
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100