one_to_all and all_to_one memcpy measurement operations are launched before "warmup" iterations are finished

In one_to_all_read and all_to_one_write testcases (both sm and ce), when stream 0 is much faster than the other streams, memcpy operations are not launched at the same time, which taints the metric collected due to overlapping "warmup" and "measurement" from separate streams.

This is due to how "warm up" runs are done in doMemcpyCore. Stream 0, if it is fastest, does not "wait" on any of the warmups to finish. Any stream other than "stream 0" has to "wait" for stream 0 to start.

So in the case that stream 0 is fastest, it finishes its warmup and immediately starts the measurement phase before stream 1,2,3... finish their warmup.

In the case that stream 0 is NOT fastest, every other stream's warmup finishes first, and then "waits" for stream 0 measurement to start, correctly aligning the measurement.

I've profiled the testcase with nsys to show how this is happening.

Incorrect alignment when stream0 is fastest and doesn't wait for 1 or 2




Correct Alignment when stream 0 and 1 are slowest, so stream 2 waits after warmup

memcpy SM GPU(row) <- GPU(column) bandwidth (GB/s)
          0         1         2         3
0     40.87     40.84     44.39     44.36    
## GPUs 0 and 1 show poor performance due to stream0 launching before others are done with warmup
## GPUs 2 and 3 show good performance, because all streams are waiting for warmup to end before launching

I've attempted my own fix by creating "warmupEndEvents" in memcpy.cpp for all streams to wait before beginning the measurement, and this fixes the issue in both cases (stream 0 fastest or slowest)
Stream 0 fastest, now correctly aligned measurement start



Stream 2 fastest, but still waits for others after warmup


Here are the results from my modified version:

memcpy SM GPU(row) <- GPU(column) bandwidth (GB/s)
           0         1         2         3
 0     44.37     44.33     44.39     44.37
## All GPUs show identical performance

Start in memcpy.cpp, especially doMemcpyCore and the one_to_all_read and all_to_one_write testcases for both sm and ce. Reproduce the stream-0-fastest case, then verify that measurement launches only after every stream's warmup completes and that the resulting bandwidth measurements are aligned across GPUs.