Identify and fix the correctness bug that happens sometimes with example 15, all_reduce kernel
Open
Nobody has claimed this yet.
bug
examples
iris
- Dominant language
- Python
- Stars
- 202
- Forks
- 47
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 4
Description
python examples/15_gemm_all_reduce_ring_based/benchmark.py --benchmark --validate --num_ranks 8
[Iris] [0/8] Validating...
[Iris] [3/8] Validating...
[Iris] [4/8] Validating...
[Iris] [1/8] Validating...
[Iris] [2/8] Validating...
[Iris] [7/8] Validating...
[Iris] [5/8] Validating...
[Iris] [6/8] Validating...
[Iris] [5/8] Max absolute difference: 414.75
[Iris] [6/8] Max absolute difference: 414.75
[Iris] [5/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [5/8] Final C validation failed.
[Iris] [6/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [6/8] Final C validation failed.
[Iris] [3/8] Max absolute difference: 414.75
[Iris] [1/8] Max absolute difference: 414.75
[Iris] [4/8] Max absolute difference: 414.75
[Iris] [3/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [1/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [3/8] Final C validation failed.
[Iris] [1/8] Final C validation failed.
[Iris] [2/8] Max absolute difference: 414.75
[Iris] [7/8] Max absolute difference: 414.75
[Iris] [4/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [4/8] Final C validation failed.
[Iris] [2/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [7/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [2/8] Final C validation failed.
[Iris] [7/8] Final C validation failed.
[Iris] [0/8] Max absolute difference: 414.75
[Iris] [0/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [0/8] Final C validation failed.
[Iris] [0/8] Validation completed
[Iris] [1/8] Validation completed
[Iris] [7/8] Validation completed
[Iris] [3/8] Validation completed
[Iris] [0/8] Benchmarking...
[Iris] [2/8] Validation completed
[Iris] [1/8] Benchmarking...
[Iris] [3/8] Benchmarking...
[Iris] [7/8] Benchmarking...
[Iris] [4/8] Validation completed
[Iris] [2/8] Benchmarking...
[Iris] [5/8] Validation completed
[Iris] [6/8] Validation completed
[Iris] [4/8] Benchmarking...
[Iris] [5/8] Benchmarking...
[Iris] [6/8] Benchmarking...
[Iris] [4/8] tile matmul + all_reduce (grid=2304): 5.232 ms 531.931 tflops
[Iris] [3/8] tile matmul + all_reduce (grid=2304): 5.232 ms 531.900 tflops
[Iris] [1/8] tile matmul + all_reduce (grid=2304): 5.229 ms 532.237 tflops
[Iris] [7/8] tile matmul + all_reduce (grid=2304): 5.231 ms 532.026 tflops
[Iris] [0/8] tile matmul + all_reduce (grid=2304): 5.233 ms 531.798 tflops
[Iris] [5/8] tile matmul + all_reduce (grid=2304): 5.235 ms 531.611 tflops
[Iris] [2/8] tile matmul + all_reduce (grid=2304): 5.235 ms 531.598 tflops
[Iris] [6/8] tile matmul + all_reduce (grid=2304): 5.094 ms 546.341 tflops
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.