NVIDIA / NVIDIA/cccl

Warpspeed scan is not clearly better than lookback scan on RTX PRO 6000

Open
#8,682 1 comment 0 reactions 1 assignee Claimed by @bernhardmgruber View on GitHub
Dominant language
C++
Stars
2.5k
Forks
487
Avg merge
2d 7h
Merged PRs (30d)
296

Description

I just compared warpspeed scan vs. the old one on an RTX PRO 6000 Server Edition (600W). The node was not entirely stable (up to 6% perf diff between runs of the same benchmark binary), but I had troubles finding a good node in general.

Here is what I measured:

baseline lookback-based scan:
```
### [0] NVIDIA RTX PRO 6000 Blackwell Server Edition

| T{ct} | OffsetT{ct} | Elements{io} | Size | Samples | CPU Time | Noise | GPU Time | Noise | Elem/s | GlobalMem BW | BWUtil |
|-------|-------------|-------------------|-------------|---------|------------|---------|------------|--------|----------|--------------|--------|
| I8 | I64 | 2^16 = 65536 | 64.000 KiB | 12x | 442.155 us | 322.11% | 8.192 us | 0.00% | 8.000G | 16.000 GB/s | 1.00% |
| I8 | I64 | 2^20 = 1048576 | 1.000 MiB | 346x | 32.913 us | 6.61% | 10.512 us | 6.62% | 99.750G | 199.501 GB/s | 12.49% |
| I8 | I64 | 2^24 = 16777216 | 16.000 MiB | 472x | 71.625 us | 2.40% | 49.884 us | 1.99% | 336.323G | 672.646 GB/s | 42.10% |
| I8 | I64 | 2^28 = 268435456 | 256.000 MiB | 776x | 537.028 us | 1.69% | 514.517 us | 1.65% | 521.723G | 1.043 TB/s | 65.31% |
| I8 | I64 | 2^32 = 4294967296 | 4.000 GiB | 484x | 7.987 ms | 3.35% | 7.962 ms | 3.36% | 539.448G | 1.079 TB/s | 67.53% |
| I16 | I64 | 2^16 = 65536 | 128.000 KiB | 98x | 28.279 us | 4.46% | 6.165 us | 4.00% | 10.631G | 42.522 GB/s | 2.66% |
| I16 | I64 | 2^20 = 1048576 | 2.000 MiB | 442x | 32.879 us | 3.46% | 11.427 us | 8.84% | 91.762G | 367.048 GB/s | 22.98% |
| I16 | I64 | 2^24 = 16777216 | 32.000 MiB | 306x | 74.002 us | 1.88% | 51.578 us | 1.60% | 325.276G | 1.301 TB/s | 81.44% |
| I16 | I64 | 2^28 = 268435456 | 512.000 MiB | 328x | 786.988 us | 0.71% | 765.287 us | 0.43% | 350.764G | 1.403 TB/s | 87.82% |
| I16 | I64 | 2^32 = 4294967296 | 8.000 GiB | 454x | 12.023 ms | 3.36% | 12.001 ms | 3.37% | 357.886G | 1.432 TB/s | 89.61% |
| I32 | I64 | 2^16 = 65536 | 256.000 KiB | 350x | 29.455 us | 3.58% | 8.189 us | 0.94% | 8.003G | 64.026 GB/s | 4.01% |
| I32 | I64 | 2^20 = 1048576 | 4.000 MiB | 12x | 35.648 us | 3.06% | 14.336 us | 0.00% | 73.143G | 585.143 GB/s | 36.63% |
| I32 | I64 | 2^24 = 16777216 | 64.000 MiB | 318x | 114.259 us | 1.43% | 91.285 us | 1.41% | 183.789G | 1.470 TB/s | 92.03% |
| I32 | I64 | 2^28 = 268435456 | 1.000 GiB | 346x | 1.492 ms | 0.15% | 1.470 ms | 0.14% | 182.576G | 1.461 TB/s | 91.43% |
| I32 | I64 | 2^32 = 4294967296 | 16.000 GiB | 540x | 23.535 ms | 1.53% | 23.513 ms | 1.53% | 182.665G | 1.461 TB/s | 91.47% |
| I64 | I64 | 2^16 = 65536 | 512.000 KiB | 328x | 29.522 us | 4.59% | 8.124 us | 2.78% | 8.066G | 129.064 GB/s | 8.08% |
| I64 | I64 | 2^20 = 1048576 | 8.000 MiB | 314x | 40.933 us | 2.79% | 19.103 us | 5.46% | 54.890G | 878.247 GB/s | 54.97% |
| I64 | I64 | 2^24 = 16777216 | 128.000 MiB | 430x | 209.106 us | 0.75% | 186.602 us | 0.99% | 89.909G | 1.439 TB/s | 90.05% |
| I64 | I64 | 2^28 = 268435456 | 2.000 GiB | 312x | 2.955 ms | 0.08% | 2.933 ms | 0.09% | 91.532G | 1.465 TB/s | 91.67% |
| I64 | I64 | 2^32 = 4294967296 | 32.000 GiB | 319x | 47.035 ms | 0.04% | 47.013 ms | 0.04% | 91.357G | 1.462 TB/s | 91.50% |
| I128 | I64 | 2^16 = 65536 | 1.000 MiB | 324x | 32.293 us | 4.20% | 11.264 us | 9.13% | 5.818G | 186.177 GB/s | 11.65% |
| I128 | I64 | 2^20 = 1048576 | 16.000 MiB | 338x | 60.945 us | 2.52% | 38.955 us | 2.57% | 26.918G | 861.364 GB/s | 53.92% |
| I128 | I64 | 2^24 = 16777216 | 256.000 MiB | 378x | 405.483 us | 0.97% | 384.191 us | 1.01% | 43.669G | 1.397 TB/s | 87.47% |
| I128 | I64 | 2^28 = 268435456 | 4.000 GiB | 514x | 5.943 ms | 0.26% | 5.921 ms | 0.22% | 45.336G | 1.451 TB/s | 90.81% |
| F32 | I64 | 2^16 = 65536 | 256.000 KiB | 362x | 29.324 us | 3.24% | 8.189 us | 0.93% | 8.003G | 64.020 GB/s | 4.01% |
| F32 | I64 | 2^20 = 1048576 | 4.000 MiB | 370x | 36.682 us | 3.23% | 14.458 us | 3.39% | 72.523G | 580.187 GB/s | 36.32% |
| F32 | I64 | 2^24 = 16777216 | 64.000 MiB | 314x | 118.293 us | 10.75% | 95.087 us | 12.89% | 176.441G | 1.412 TB/s | 88.35% |
| F32 | I64 | 2^28 = 268435456 | 1.000 GiB | 352x | 1.493 ms | 0.17% | 1.470 ms | 0.14% | 182.598G | 1.461 TB/s | 91.44% |
| F32 | I64 | 2^32 = 4294967296 | 16.000 GiB | 484x | 23.539 ms | 1.73% | 23.516 ms | 1.73% | 182.640G | 1.461 TB/s | 91.46% |
| F64 | I64 | 2^16 = 65536 | 512.000 KiB | 490x | 31.106 us | 4.30% | 9.781 us | 8.76% | 6.700G | 107.207 GB/s | 6.71% |
| F64 | I64 | 2^20 = 1048576 | 8.000 MiB | 540x | 42.011 us | 2.42% | 20.489 us | 0.76% | 51.177G | 818.828 GB/s | 51.25% |
| F64 | I64 | 2^24 = 16777216 | 128.000 MiB | 536x | 207.358 us | 0.83% | 184.807 us | 0.80% | 90.782G | 1.453 TB/s | 90.92% |
| F64 | I64 | 2^28 = 268435456 | 2.000 GiB | 416x | 2.959 ms | 0.09% | 2.937 ms | 0.09% | 91.403G | 1.462 TB/s | 91.54% |
| F64 | I64 | 2^32 = 4294967296 | 32.000 GiB | 318x | 47.082 ms | 0.03% | 47.059 ms | 0.03% | 91.267G | 1.460 TB/s | 91.41% |
| C32 | I64 | 2^16 = 65536 | 512.000 KiB | 454x | 29.216 us | 4.52% | 7.735 us | 10.69% | 8.473G | 135.563 GB/s | 8.49% |
| C32 | I64 | 2^20 = 1048576 | 8.000 MiB | 340x | 39.947 us | 3.07% | 18.131 us | 4.64% | 57.832G | 925.318 GB/s | 57.92% |
| C32 | I64 | 2^24 = 16777216 | 128.000 MiB | 368x | 209.081 us | 0.69% | 186.589 us | 0.57% | 89.916G | 1.439 TB/s | 90.05% |
| C32 | I64 | 2^28 = 268435456 | 2.000 GiB | 414x | 2.963 ms | 0.15% | 2.940 ms | 0.12% | 91.290G | 1.461 TB/s | 91.43% |
| C32 | I64 | 2^32 = 4294967296 | 32.000 GiB | 318x | 47.154 ms | 0.03% | 47.131 ms | 0.03% | 91.129G | 1.458 TB/s | 91.27% |
```

new warp-specialized TMA scan (warpspeed)
```
### [0] NVIDIA RTX PRO 6000 Blackwell Server Edition

| T{ct} | OffsetT{ct} | Elements{io} | Size | Samples | CPU Time | Noise | GPU Time | Noise | Elem/s | GlobalMem BW | BWUtil |
|-------|-------------|-------------------|-------------|---------|------------|---------|------------|--------|----------|--------------|--------|
| I8 | I64 | 2^16 = 65536 | 64.000 KiB | 402x | 41.508 us | 600.80% | 6.486 us | 11.76% | 10.104G | 20.208 GB/s | 1.26% |
| I8 | I64 | 2^20 = 1048576 | 1.000 MiB | 372x | 38.053 us | 5.05% | 9.100 us | 11.27% | 115.234G | 230.468 GB/s | 14.43% |
| I8 | I64 | 2^24 = 16777216 | 16.000 MiB | 482x | 60.691 us | 3.89% | 30.952 us | 2.11% | 542.039G | 1.084 TB/s | 67.86% |
| I8 | I64 | 2^28 = 268435456 | 256.000 MiB | 378x | 399.714 us | 0.63% | 369.830 us | 0.67% | 725.836G | 1.452 TB/s | 90.87% |
| I8 | I64 | 2^32 = 4294967296 | 4.000 GiB | 400x | 5.944 ms | 5.59% | 5.919 ms | 5.59% | 725.627G | 1.451 TB/s | 90.84% |
| I16 | I64 | 2^16 = 65536 | 128.000 KiB | 308x | 31.116 us | 10.94% | 8.192 us | 0.23% | 8.000G | 31.999 GB/s | 2.00% |
| I16 | I64 | 2^20 = 1048576 | 2.000 MiB | 398x | 32.804 us | 5.12% | 10.680 us | 7.93% | 98.182G | 392.730 GB/s | 24.58% |
| I16 | I64 | 2^24 = 16777216 | 32.000 MiB | 346x | 69.286 us | 2.54% | 45.311 us | 1.61% | 370.272G | 1.481 TB/s | 92.71% |
| I16 | I64 | 2^28 = 268435456 | 512.000 MiB | 328x | 759.080 us | 0.32% | 735.946 us | 0.21% | 364.749G | 1.459 TB/s | 91.33% |
| I16 | I64 | 2^32 = 4294967296 | 8.000 GiB | 382x | 11.849 ms | 2.72% | 11.826 ms | 2.73% | 363.178G | 1.453 TB/s | 90.93% |
| I32 | I64 | 2^16 = 65536 | 256.000 KiB | 284x | 30.656 us | 4.12% | 8.198 us | 1.57% | 7.994G | 63.953 GB/s | 4.00% |
| I32 | I64 | 2^20 = 1048576 | 4.000 MiB | 12x | 38.524 us | 5.60% | 14.336 us | 0.00% | 73.143G | 585.143 GB/s | 36.63% |
| I32 | I64 | 2^24 = 16777216 | 64.000 MiB | 318x | 114.046 us | 2.01% | 90.622 us | 1.25% | 185.134G | 1.481 TB/s | 92.71% |
| I32 | I64 | 2^28 = 268435456 | 1.000 GiB | 320x | 1.495 ms | 0.13% | 1.472 ms | 0.13% | 182.389G | 1.459 TB/s | 91.33% |
| I32 | I64 | 2^32 = 4294967296 | 16.000 GiB | 498x | 23.628 ms | 1.53% | 23.605 ms | 1.53% | 181.951G | 1.456 TB/s | 91.11% |
| I64 | I64 | 2^16 = 65536 | 512.000 KiB | 358x | 30.521 us | 4.58% | 8.211 us | 2.88% | 7.982G | 127.707 GB/s | 7.99% |
| I64 | I64 | 2^20 = 1048576 | 8.000 MiB | 352x | 40.665 us | 2.72% | 18.447 us | 1.02% | 56.842G | 909.469 GB/s | 56.93% |
| I64 | I64 | 2^24 = 16777216 | 128.000 MiB | 304x | 206.875 us | 1.23% | 183.376 us | 0.68% | 91.491G | 1.464 TB/s | 91.63% |
| I64 | I64 | 2^28 = 268435456 | 2.000 GiB | 370x | 2.969 ms | 0.11% | 2.946 ms | 0.11% | 91.114G | 1.458 TB/s | 91.25% |
| I64 | I64 | 2^32 = 4294967296 | 32.000 GiB | 317x | 47.228 ms | 0.05% | 47.204 ms | 0.05% | 90.988G | 1.456 TB/s | 91.13% |
| I128 | I64 | 2^16 = 65536 | 1.000 MiB | 470x | 31.975 us | 3.95% | 10.223 us | 1.67% | 6.411G | 205.148 GB/s | 12.84% |
| I128 | I64 | 2^20 = 1048576 | 16.000 MiB | 622x | 62.357 us | 5.32% | 40.653 us | 6.67% | 25.794G | 825.396 GB/s | 51.67% |
| I128 | I64 | 2^24 = 16777216 | 256.000 MiB | 414x | 411.091 us | 0.59% | 389.416 us | 0.51% | 43.083G | 1.379 TB/s | 86.30% |
| I128 | I64 | 2^28 = 268435456 | 4.000 GiB | 336x | 5.967 ms | 0.10% | 5.945 ms | 0.09% | 45.152G | 1.445 TB/s | 90.44% |
| F32 | I64 | 2^16 = 65536 | 256.000 KiB | 12x | 30.464 us | 3.74% | 8.192 us | 0.00% | 8.000G | 64.000 GB/s | 4.01% |
| F32 | I64 | 2^20 = 1048576 | 4.000 MiB | 480x | 37.827 us | 3.74% | 15.208 us | 6.88% | 68.949G | 551.592 GB/s | 34.53% |
| F32 | I64 | 2^24 = 16777216 | 64.000 MiB | 420x | 122.337 us | 9.42% | 98.906 us | 11.38% | 169.627G | 1.357 TB/s | 84.94% |
| F32 | I64 | 2^28 = 268435456 | 1.000 GiB | 326x | 1.494 ms | 0.20% | 1.470 ms | 0.12% | 182.548G | 1.460 TB/s | 91.41% |
| F32 | I64 | 2^32 = 4294967296 | 16.000 GiB | 488x | 23.620 ms | 1.48% | 23.597 ms | 1.48% | 182.015G | 1.456 TB/s | 91.15% |
| F64 | I64 | 2^16 = 65536 | 512.000 KiB | 376x | 32.761 us | 4.55% | 10.465 us | 6.13% | 6.262G | 100.197 GB/s | 6.27% |
| F64 | I64 | 2^20 = 1048576 | 8.000 MiB | 322x | 44.113 us | 3.29% | 21.736 us | 4.71% | 48.242G | 771.872 GB/s | 48.32% |
| F64 | I64 | 2^24 = 16777216 | 128.000 MiB | 322x | 209.535 us | 4.93% | 185.907 us | 0.76% | 90.245G | 1.444 TB/s | 90.38% |
| F64 | I64 | 2^28 = 268435456 | 2.000 GiB | 380x | 2.970 ms | 0.12% | 2.947 ms | 0.10% | 91.078G | 1.457 TB/s | 91.22% |
| F64 | I64 | 2^32 = 4294967296 | 32.000 GiB | 318x | 47.204 ms | 0.03% | 47.181 ms | 0.03% | 91.032G | 1.457 TB/s | 91.17% |
| C32 | I64 | 2^16 = 65536 | 512.000 KiB | 348x | 30.527 us | 15.22% | 8.208 us | 2.59% | 7.985G | 127.756 GB/s | 8.00% |
| C32 | I64 | 2^20 = 1048576 | 8.000 MiB | 380x | 41.627 us | 3.11% | 18.625 us | 3.67% | 56.300G | 900.798 GB/s | 56.39% |
| C32 | I64 | 2^24 = 16777216 | 128.000 MiB | 310x | 207.046 us | 0.75% | 183.330 us | 0.66% | 91.514G | 1.464 TB/s | 91.65% |
| C32 | I64 | 2^28 = 268435456 | 2.000 GiB | 418x | 2.969 ms | 0.11% | 2.946 ms | 0.11% | 91.121G | 1.458 TB/s | 91.26% |
| C32 | I64 | 2^32 = 4294967296 | 32.000 GiB | 317x | 47.233 ms | 0.04% | 47.209 ms | 0.04% | 90.979G | 1.456 TB/s | 91.12% |
```

Diff:
```
## [0] NVIDIA RTX PRO 6000 Blackwell Server Edition

| T{ct} | OffsetT{ct} | Elements{io} | Ref Time | Ref Noise | Cmp Time | Cmp Noise | Diff | %Diff | Status |
|---------|---------------|----------------|------------|-------------|------------|-------------|--------------|---------|----------|
| I8 | I64 | 2^16 | 8.192 us | 0.00% | 6.486 us | 11.76% | -1.706 us | -20.82% | 🟒 FAST |
| I8 | I64 | 2^20 | 10.512 us | 6.62% | 9.100 us | 11.27% | -1.412 us | -13.44% | 🟒 FAST |
| I8 | I64 | 2^24 | 49.884 us | 1.99% | 30.952 us | 2.11% | -18.932 us | -37.95% | 🟒 FAST |
| I8 | I64 | 2^28 | 514.517 us | 1.65% | 369.830 us | 0.67% | -144.687 us | -28.12% | 🟒 FAST |
| I8 | I64 | 2^32 | 7.962 ms | 3.36% | 5.919 ms | 5.59% | -2042.812 us | -25.66% | 🟒 FAST |
| I16 | I64 | 2^16 | 6.165 us | 4.00% | 8.192 us | 0.23% | 2.027 us | 32.88% | πŸ”΄ SLOW |
| I16 | I64 | 2^20 | 11.427 us | 8.84% | 10.680 us | 7.93% | -0.747 us | -6.54% | πŸ”΅ SAME |
| I16 | I64 | 2^24 | 51.578 us | 1.60% | 45.311 us | 1.61% | -6.268 us | -12.15% | 🟒 FAST |
| I16 | I64 | 2^28 | 765.287 us | 0.43% | 735.946 us | 0.21% | -29.341 us | -3.83% | 🟒 FAST |
| I16 | I64 | 2^32 | 12.001 ms | 3.37% | 11.826 ms | 2.73% | -174.876 us | -1.46% | πŸ”΅ SAME |
| I32 | I64 | 2^16 | 8.189 us | 0.94% | 8.198 us | 1.57% | 0.009 us | 0.11% | πŸ”΅ SAME |
| I32 | I64 | 2^20 | 14.336 us | 0.00% | 14.336 us | 0.00% | 0.000 us | 0.00% | 🟑 ???? |
| I32 | I64 | 2^24 | 91.285 us | 1.41% | 90.622 us | 1.25% | -0.663 us | -0.73% | πŸ”΅ SAME |
| I32 | I64 | 2^28 | 1.470 ms | 0.14% | 1.472 ms | 0.13% | 1.510 us | 0.10% | πŸ”΅ SAME |
| I32 | I64 | 2^32 | 23.513 ms | 1.53% | 23.605 ms | 1.53% | 92.208 us | 0.39% | πŸ”΅ SAME |
| I64 | I64 | 2^16 | 8.124 us | 2.78% | 8.211 us | 2.88% | 0.086 us | 1.06% | πŸ”΅ SAME |
| I64 | I64 | 2^20 | 19.103 us | 5.46% | 18.447 us | 1.02% | -0.656 us | -3.43% | 🟒 FAST |
| I64 | I64 | 2^24 | 186.602 us | 0.99% | 183.376 us | 0.68% | -3.226 us | -1.73% | 🟒 FAST |
| I64 | I64 | 2^28 | 2.933 ms | 0.09% | 2.946 ms | 0.11% | 13.448 us | 0.46% | πŸ”΄ SLOW |
| I64 | I64 | 2^32 | 47.013 ms | 0.04% | 47.204 ms | 0.05% | 190.957 us | 0.41% | πŸ”΄ SLOW |
| I128 | I64 | 2^16 | 11.264 us | 9.13% | 10.223 us | 1.67% | -1.042 us | -9.25% | 🟒 FAST |
| I128 | I64 | 2^20 | 38.955 us | 2.57% | 40.653 us | 6.67% | 1.698 us | 4.36% | πŸ”΄ SLOW |
| I128 | I64 | 2^24 | 384.191 us | 1.01% | 389.416 us | 0.51% | 5.224 us | 1.36% | πŸ”΄ SLOW |
| I128 | I64 | 2^28 | 5.921 ms | 0.22% | 5.945 ms | 0.09% | 24.221 us | 0.41% | πŸ”΄ SLOW |
| F32 | I64 | 2^16 | 8.189 us | 0.93% | 8.192 us | 0.00% | 0.003 us | 0.03% | πŸ”΄ SLOW |
| F32 | I64 | 2^20 | 14.458 us | 3.39% | 15.208 us | 6.88% | 0.750 us | 5.18% | πŸ”΄ SLOW |
| F32 | I64 | 2^24 | 95.087 us | 12.89% | 98.906 us | 11.38% | 3.820 us | 4.02% | πŸ”΅ SAME |
| F32 | I64 | 2^28 | 1.470 ms | 0.14% | 1.470 ms | 0.12% | 0.402 us | 0.03% | πŸ”΅ SAME |
| F32 | I64 | 2^32 | 23.516 ms | 1.73% | 23.597 ms | 1.48% | 80.832 us | 0.34% | πŸ”΅ SAME |
| F64 | I64 | 2^16 | 9.781 us | 8.76% | 10.465 us | 6.13% | 0.684 us | 7.00% | πŸ”΄ SLOW |
| F64 | I64 | 2^20 | 20.489 us | 0.76% | 21.736 us | 4.71% | 1.246 us | 6.08% | πŸ”΄ SLOW |
| F64 | I64 | 2^24 | 184.807 us | 0.80% | 185.907 us | 0.76% | 1.100 us | 0.60% | πŸ”΅ SAME |
| F64 | I64 | 2^28 | 2.937 ms | 0.09% | 2.947 ms | 0.10% | 10.483 us | 0.36% | πŸ”΄ SLOW |
| F64 | I64 | 2^32 | 47.059 ms | 0.03% | 47.181 ms | 0.03% | 121.481 us | 0.26% | πŸ”΄ SLOW |
| C32 | I64 | 2^16 | 7.735 us | 10.69% | 8.208 us | 2.59% | 0.473 us | 6.11% | πŸ”΄ SLOW |
| C32 | I64 | 2^20 | 18.131 us | 4.64% | 18.625 us | 3.67% | 0.494 us | 2.72% | πŸ”΅ SAME |
| C32 | I64 | 2^24 | 186.589 us | 0.57% | 183.330 us | 0.66% | -3.258 us | -1.75% | 🟒 FAST |
| C32 | I64 | 2^28 | 2.940 ms | 0.12% | 2.946 ms | 0.11% | 5.446 us | 0.19% | πŸ”΄ SLOW |
| C32 | I64 | 2^32 | 47.131 ms | 0.03% | 47.209 ms | 0.04% | 77.717 us | 0.16% | πŸ”΄ SLOW |
```
So while there is a clear benefit on I8, it's less clear for other data types. We are however still waiting for tuning values for SM120. Before branching CCCL 3.4, we should revisit this again and potentially disable warpspeed scan for some workloads.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.