Performance config for 3080

Hello NVIDIA,

I'm trying to tune the performance of libntc on my 3080, here is some progress. The config is windows 11, RTX 3080, CUDA 12.9 and driver 581.29, Nsight Compute seems not work on 590.26.

The command I use

"bin/windows-x64/ntc-cli" --loadManifest=./assets/materials/MetalPlates013/Manifest.json -c -b 3 --gridLearningRate=0.1 --kPixelsPerBatch=64 --networkLearningRate=0.005 --randomSeed=3 --stepsPerIteration=1000 --trainingSteps=100000 -o nvidia.ntc

Nsight reports “Low compute throughput and memory bandwidth utilization” and "Tail effect" in the report.

looks like that the workload can be increased in the kernel, tail effect might be caused by the grid size cannot be divided by SM count evenly, kPixelsPerBatch * 1024 / WARPS_PER_TBLOCK / warpSize / 68 = 7.53

So I just adjust Y_ITERS to 4 to increase the workload in each thread, this also can relieve the tail effect as grid size becomes to kPixelsPerBatch * 1024 / WARPS_PER_TBLOCK / warpSize / 4 / 68 = 1.88, 4 seems to be an ideal config for my 3080.

The performance when Y_ITERS is 1 is 0.57 ms/step at the beginning of training.

The performance when Y_ITERS is 4 is 0.55 ms/step at the beginning of training. Even more improvement in the last epoch, reduce from 0.6553 ms/step to 0.6121 ms/step, hopefully this config can help a little as I don't have enough test case, do you have any recommended case?

And one bad news for CUDA 13.1, it isn't mentioned in the README, but there are many issues, 581.29 works but 590.26 doesn't work on CUDA 13.1. CUDA 13.1 is slower than 12.9 for training, it will take more than 1ms at the beginning of training. Hopefully you can find the root cause, probably it's caused by the driver.

Best Regards
Hui

Reproduce the bin/windows-x64/ntc-cli command with the stated manifest and compare Y_ITERS values of 1 and 4 on the RTX 3080. Use Nsight Compute where available to examine the reported throughput and tail effects, then compare CUDA 12.9 and 13.1 with the listed drivers. Done means confirming the recommended configuration or identifying and documenting the CUDA 13.1 performance cause.