ROCm / ROCm/iris

Identify and fix the correctness bug that happens sometimes with example 15, all_reduce kernel

Open
#226 0 comments 1 reaction 1 assignee View on GitHub

Nobody has claimed this yet.

bug examples iris
Dominant language
Python
Stars
202
Forks
47
Avg merge
6d 11h
Merged PRs (30d)
4

Description

python examples/15_gemm_all_reduce_ring_based/benchmark.py --benchmark --validate --num_ranks 8
[Iris] [0/8] Validating...
[Iris] [3/8] Validating...
[Iris] [4/8] Validating...
[Iris] [1/8] Validating...
[Iris] [2/8] Validating...
[Iris] [7/8] Validating...
[Iris] [5/8] Validating...
[Iris] [6/8] Validating...
[Iris] [5/8] Max absolute difference: 414.75
[Iris] [6/8] Max absolute difference: 414.75
[Iris] [5/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [5/8] Final C validation failed.
[Iris] [6/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [6/8] Final C validation failed.
[Iris] [3/8] Max absolute difference: 414.75
[Iris] [1/8] Max absolute difference: 414.75
[Iris] [4/8] Max absolute difference: 414.75
[Iris] [3/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [1/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [3/8] Final C validation failed.
[Iris] [1/8] Final C validation failed.
[Iris] [2/8] Max absolute difference: 414.75
[Iris] [7/8] Max absolute difference: 414.75
[Iris] [4/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [4/8] Final C validation failed.
[Iris] [2/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [7/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [2/8] Final C validation failed.
[Iris] [7/8] Final C validation failed.
[Iris] [0/8] Max absolute difference: 414.75
[Iris] [0/8] Mismatch at index (64, 1472): C=8.0546875, expected=-46.34375
[Iris] [0/8] Final C validation failed.
[Iris] [0/8] Validation completed
[Iris] [1/8] Validation completed
[Iris] [7/8] Validation completed
[Iris] [3/8] Validation completed
[Iris] [0/8] Benchmarking...
[Iris] [2/8] Validation completed
[Iris] [1/8] Benchmarking...
[Iris] [3/8] Benchmarking...
[Iris] [7/8] Benchmarking...
[Iris] [4/8] Validation completed
[Iris] [2/8] Benchmarking...
[Iris] [5/8] Validation completed
[Iris] [6/8] Validation completed
[Iris] [4/8] Benchmarking...
[Iris] [5/8] Benchmarking...
[Iris] [6/8] Benchmarking...
[Iris] [4/8] tile matmul + all_reduce (grid=2304): 5.232 ms  531.931 tflops
[Iris] [3/8] tile matmul + all_reduce (grid=2304): 5.232 ms  531.900 tflops
[Iris] [1/8] tile matmul + all_reduce (grid=2304): 5.229 ms  532.237 tflops
[Iris] [7/8] tile matmul + all_reduce (grid=2304): 5.231 ms  532.026 tflops
[Iris] [0/8] tile matmul + all_reduce (grid=2304): 5.233 ms  531.798 tflops
[Iris] [5/8] tile matmul + all_reduce (grid=2304): 5.235 ms  531.611 tflops
[Iris] [2/8] tile matmul + all_reduce (grid=2304): 5.235 ms  531.598 tflops
[Iris] [6/8] tile matmul + all_reduce (grid=2304): 5.094 ms  546.341 tflops

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.