Lightning-AI / Lightning-AI/lightning-thunder

test_core_vs_torch_consistency_pow has flaky variants

Open
#1,130 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

nvfuser testing thunderfx
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

Update:

A closely related test also failed for with the int8 dtype:

```
2024-09-11T14:23:20.0323791Z =================================== FAILURES ===================================
2024-09-11T14:23:20.0324877Z ______ test_core_vs_torch_consistency_pow_torch_cuda_thunder.dtypes.int8 _______
2024-09-11T14:23:20.0325756Z [gw5] linux -- Python 3.10.12 /usr/bin/python3.10
2024-09-11T14:23:20.0326075Z
2024-09-11T14:23:20.0326629Z def test():
2024-09-11T14:23:20.0339274Z > result = template(opinfo, device_str, dtype, executor, comp)
2024-09-11T14:23:20.0340892Z
2024-09-11T14:23:20.0342094Z thunder/tests/framework.py:330:
2024-09-11T14:23:20.0342801Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2024-09-11T14:23:20.0343451Z thunder/tests/test_ops.py:75: in test_core_vs_torch_consistency
2024-09-11T14:23:20.0343930Z result = run_snippet(
2024-09-11T14:23:20.0344407Z thunder/tests/framework.py:537: in run_snippet
2024-09-11T14:23:20.0344731Z raise ex
2024-09-11T14:23:20.0345128Z thunder/tests/framework.py:529: in run_snippet
2024-09-11T14:23:20.0345459Z snippet(*args, **kwargs)
2024-09-11T14:23:20.0360378Z thunder/tests/test_ops.py:37: in snippet_torch_consistency
2024-09-11T14:23:20.0360678Z thunder_result = op(*sample.args, **sample.kwargs)
2024-09-11T14:23:20.0360997Z thunder/__init__.py:722: in fn_
2024-09-11T14:23:20.0361245Z result = cache_entry.computation_fn(*inps)
2024-09-11T14:23:20.0361611Z /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:116: in decorate_context
2024-09-11T14:23:20.0361892Z return func(*args, **kwargs)
2024-09-11T14:23:20.0362254Z /usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py:44: in decorate_autocast
2024-09-11T14:23:20.0362528Z return func(*args, **kwargs)
2024-09-11T14:23:20.0362912Z /usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py:44: in decorate_autocast
2024-09-11T14:23:20.0363365Z return func(*args, **kwargs)
2024-09-11T14:23:20.0363808Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2024-09-11T14:23:20.0363955Z
2024-09-11T14:23:20.0364337Z t_0 = tensor([[ 8, 9, 6, 6],
2024-09-11T14:23:20.0364603Z [ 1, -8, -7, -3],
2024-09-11T14:23:20.0364813Z [ 1, 8, 5, -7],
2024-09-11T14:23:20.0365116Z [-6, 6, -9, -8]], device='cuda:0', dtype=torch.int8)
2024-09-11T14:23:20.0367017Z t_1 = tensor(-1, dtype=torch.int8)
2024-09-11T14:23:20.0367128Z
2024-09-11T14:23:20.0367328Z @torch.no_grad()
2024-09-11T14:23:20.0367534Z @no_autocast
2024-09-11T14:23:20.0367746Z def computation(t_0, t_1):
2024-09-11T14:23:20.0367976Z # t_0: "cuda:0 i8[4, 4]"
2024-09-11T14:23:20.0368188Z # t_1: "cpu i8[]"
2024-09-11T14:23:20.0368416Z > t0 = torch.pow(t_0, t_1) # t0: "cuda:0 i8[4, 4]"
2024-09-11T14:23:20.0368750Z E RuntimeError: "reciprocal_cuda" not implemented for 'Char'
2024-09-11T14:23:20.0368886Z
2024-09-11T14:23:20.0369184Z thunder.computation_3876:10: RuntimeError

See https://dev.azure.com/Lightning-AI/lightning/_build/results?buildId=214380&view=logs&j=5b0799f7-725e-5b16-9b83-c0a5a25d03f0&t=97651ec4-0b0f-5455-bbb5-3c30427a0a7e
```

Original snippet showing a failure with the nvFuser consistency test and int64

```
2024-09-06T16:29:22.1460811Z =================================== FAILURES ===================================
2024-09-06T16:29:22.1461442Z _____ test_core_vs_torch_consistency_pow_nvfuser_cuda_thunder.dtypes.int64 _____
2024-09-06T16:29:22.1462171Z [gw3] linux -- Python 3.10.12 /usr/bin/python3.10
2024-09-06T16:29:22.1462423Z
2024-09-06T16:29:22.1471353Z def test():
2024-09-06T16:29:22.1473177Z > result = template(opinfo, device_str, dtype, executor, comp)
2024-09-06T16:29:22.1473474Z
2024-09-06T16:29:22.1474376Z thunder/tests/framework.py:342:
2024-09-06T16:29:22.1474873Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2024-09-06T16:29:22.1475518Z thunder/tests/test_ops.py:76: in test_core_vs_torch_consistency
2024-09-06T16:29:22.1476009Z result = run_snippet(
2024-09-06T16:29:22.1476444Z thunder/tests/framework.py:549: in run_snippet
2024-09-06T16:29:22.1481224Z raise ex
2024-09-06T16:29:22.1482057Z thunder/tests/framework.py:541: in run_snippet
2024-09-06T16:29:22.1482559Z snippet(*args, **kwargs)
2024-09-06T16:29:22.1483013Z thunder/tests/test_ops.py:44: in snippet_torch_consistency
2024-09-06T16:29:22.1483358Z comp(thunder_result, torch_result)
2024-09-06T16:29:22.1483686Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2024-09-06T16:29:22.1483871Z
2024-09-06T16:29:22.1484129Z a = tensor([[0, 0, 0, 0],
2024-09-06T16:29:22.1491876Z [0, 0, 0, 0],
2024-09-06T16:29:22.1492687Z [0, 0, 0, 0],
2024-09-06T16:29:22.1493419Z [0, 0, 0, 0]], device='cuda:0')
2024-09-06T16:29:22.1493691Z b = tensor([[ 0, 0, 0,
2024-09-06T16:29:22.1494352Z 0],
2024-09-06T16:29:22.1494567Z [ ... [ 0, 0, 0,
2024-09-06T16:29:22.1494864Z 0]], device='cuda:0')
2024-09-06T16:29:22.1494998Z
2024-09-06T16:29:22.1495210Z > lambda a, b: comp(a, b, equal_nan=True),
2024-09-06T16:29:22.1495419Z )
2024-09-06T16:29:22.1495702Z E AssertionError: Tensor-likes are not equal!
2024-09-06T16:29:22.1495969Z E 
2024-09-06T16:29:22.1496968Z E Mismatched elements: 1 / 16 (6.2%)
2024-09-06T16:29:22.1497724Z E Greatest absolute difference: 9223372036854775807 at index (2, 3)
2024-09-06T16:29:22.1498248Z E Greatest relative difference: 1.0 at index (2, 3)
2024-09-06T16:29:22.1498422Z
2024-09-06T16:29:22.1498774Z thunder/tests/test_ops.py:84: AssertionError
```

cc @tfogal @apaz-cli

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in thunder/tests/test_ops.py at test_core_vs_torch_consistency and follow the referenced helpers in thunder/tests/framework.py. Reproduce the pow cases for the nvFuser and Torch executors with CUDA int64 and int8 inputs. Done means these consistency tests no longer fail or raise the reported unsupported-dtype error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.