Lightning-AI / Lightning-AI/lightning-thunder
test_core_vs_torch_consistency_pow has flaky variants
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
Update:
A closely related test also failed for with the int8 dtype:
```
2024-09-11T14:23:20.0323791Z =================================== FAILURES ===================================
2024-09-11T14:23:20.0324877Z [31m[1m______ test_core_vs_torch_consistency_pow_torch_cuda_thunder.dtypes.int8 _______[0m
2024-09-11T14:23:20.0325756Z [gw5] linux -- Python 3.10.12 /usr/bin/python3.10
2024-09-11T14:23:20.0326075Z
2024-09-11T14:23:20.0326629Z def test():
2024-09-11T14:23:20.0339274Z > result = template(opinfo, device_str, dtype, executor, comp)
2024-09-11T14:23:20.0340892Z
2024-09-11T14:23:20.0342094Z [1m[31mthunder/tests/framework.py[0m:330:
2024-09-11T14:23:20.0342801Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2024-09-11T14:23:20.0343451Z [1m[31mthunder/tests/test_ops.py[0m:75: in test_core_vs_torch_consistency
2024-09-11T14:23:20.0343930Z result = run_snippet(
2024-09-11T14:23:20.0344407Z [1m[31mthunder/tests/framework.py[0m:537: in run_snippet
2024-09-11T14:23:20.0344731Z raise ex
2024-09-11T14:23:20.0345128Z [1m[31mthunder/tests/framework.py[0m:529: in run_snippet
2024-09-11T14:23:20.0345459Z snippet(*args, **kwargs)
2024-09-11T14:23:20.0360378Z [1m[31mthunder/tests/test_ops.py[0m:37: in snippet_torch_consistency
2024-09-11T14:23:20.0360678Z thunder_result = op(*sample.args, **sample.kwargs)
2024-09-11T14:23:20.0360997Z [1m[31mthunder/__init__.py[0m:722: in fn_
2024-09-11T14:23:20.0361245Z result = cache_entry.computation_fn(*inps)
2024-09-11T14:23:20.0361611Z [1m[31m/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py[0m:116: in decorate_context
2024-09-11T14:23:20.0361892Z return func(*args, **kwargs)
2024-09-11T14:23:20.0362254Z [1m[31m/usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py[0m:44: in decorate_autocast
2024-09-11T14:23:20.0362528Z return func(*args, **kwargs)
2024-09-11T14:23:20.0362912Z [1m[31m/usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py[0m:44: in decorate_autocast
2024-09-11T14:23:20.0363365Z return func(*args, **kwargs)
2024-09-11T14:23:20.0363808Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2024-09-11T14:23:20.0363955Z
2024-09-11T14:23:20.0364337Z t_0 = tensor([[ 8, 9, 6, 6],
2024-09-11T14:23:20.0364603Z [ 1, -8, -7, -3],
2024-09-11T14:23:20.0364813Z [ 1, 8, 5, -7],
2024-09-11T14:23:20.0365116Z [-6, 6, -9, -8]], device='cuda:0', dtype=torch.int8)
2024-09-11T14:23:20.0367017Z t_1 = tensor(-1, dtype=torch.int8)
2024-09-11T14:23:20.0367128Z
2024-09-11T14:23:20.0367328Z @torch.no_grad()
2024-09-11T14:23:20.0367534Z @no_autocast
2024-09-11T14:23:20.0367746Z def computation(t_0, t_1):
2024-09-11T14:23:20.0367976Z # t_0: "cuda:0 i8[4, 4]"
2024-09-11T14:23:20.0368188Z # t_1: "cpu i8[]"
2024-09-11T14:23:20.0368416Z > t0 = torch.pow(t_0, t_1) # t0: "cuda:0 i8[4, 4]"
2024-09-11T14:23:20.0368750Z [1m[31mE RuntimeError: "reciprocal_cuda" not implemented for 'Char'[0m
2024-09-11T14:23:20.0368886Z
2024-09-11T14:23:20.0369184Z [1m[31mthunder.computation_3876[0m:10: RuntimeError
See https://dev.azure.com/Lightning-AI/lightning/_build/results?buildId=214380&view=logs&j=5b0799f7-725e-5b16-9b83-c0a5a25d03f0&t=97651ec4-0b0f-5455-bbb5-3c30427a0a7e
```
Original snippet showing a failure with the nvFuser consistency test and int64
```
2024-09-06T16:29:22.1460811Z =================================== FAILURES ===================================
2024-09-06T16:29:22.1461442Z [31m[1m_____ test_core_vs_torch_consistency_pow_nvfuser_cuda_thunder.dtypes.int64 _____[0m
2024-09-06T16:29:22.1462171Z [gw3] linux -- Python 3.10.12 /usr/bin/python3.10
2024-09-06T16:29:22.1462423Z
2024-09-06T16:29:22.1471353Z def test():
2024-09-06T16:29:22.1473177Z > result = template(opinfo, device_str, dtype, executor, comp)
2024-09-06T16:29:22.1473474Z
2024-09-06T16:29:22.1474376Z [1m[31mthunder/tests/framework.py[0m:342:
2024-09-06T16:29:22.1474873Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2024-09-06T16:29:22.1475518Z [1m[31mthunder/tests/test_ops.py[0m:76: in test_core_vs_torch_consistency
2024-09-06T16:29:22.1476009Z result = run_snippet(
2024-09-06T16:29:22.1476444Z [1m[31mthunder/tests/framework.py[0m:549: in run_snippet
2024-09-06T16:29:22.1481224Z raise ex
2024-09-06T16:29:22.1482057Z [1m[31mthunder/tests/framework.py[0m:541: in run_snippet
2024-09-06T16:29:22.1482559Z snippet(*args, **kwargs)
2024-09-06T16:29:22.1483013Z [1m[31mthunder/tests/test_ops.py[0m:44: in snippet_torch_consistency
2024-09-06T16:29:22.1483358Z comp(thunder_result, torch_result)
2024-09-06T16:29:22.1483686Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2024-09-06T16:29:22.1483871Z
2024-09-06T16:29:22.1484129Z a = tensor([[0, 0, 0, 0],
2024-09-06T16:29:22.1491876Z [0, 0, 0, 0],
2024-09-06T16:29:22.1492687Z [0, 0, 0, 0],
2024-09-06T16:29:22.1493419Z [0, 0, 0, 0]], device='cuda:0')
2024-09-06T16:29:22.1493691Z b = tensor([[ 0, 0, 0,
2024-09-06T16:29:22.1494352Z 0],
2024-09-06T16:29:22.1494567Z [ ... [ 0, 0, 0,
2024-09-06T16:29:22.1494864Z 0]], device='cuda:0')
2024-09-06T16:29:22.1494998Z
2024-09-06T16:29:22.1495210Z > lambda a, b: comp(a, b, equal_nan=True),
2024-09-06T16:29:22.1495419Z )
2024-09-06T16:29:22.1495702Z [1m[31mE AssertionError: Tensor-likes are not equal![0m
2024-09-06T16:29:22.1495969Z [1m[31mE [0m
2024-09-06T16:29:22.1496968Z [1m[31mE Mismatched elements: 1 / 16 (6.2%)[0m
2024-09-06T16:29:22.1497724Z [1m[31mE Greatest absolute difference: 9223372036854775807 at index (2, 3)[0m
2024-09-06T16:29:22.1498248Z [1m[31mE Greatest relative difference: 1.0 at index (2, 3)[0m
2024-09-06T16:29:22.1498422Z
2024-09-06T16:29:22.1498774Z [1m[31mthunder/tests/test_ops.py[0m:84: AssertionError
```
cc @tfogal @apaz-cli
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in thunder/tests/test_ops.py at test_core_vs_torch_consistency and follow the referenced helpers in thunder/tests/framework.py. Reproduce the pow cases for the nvFuser and Torch executors with CUDA int64 and int8 inputs. Done means these consistency tests no longer fail or raise the reported unsupported-dtype error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100