NVIDIA / NVIDIA/cccl

thrust::universal_vector push_back is very slow

Open
#809 0 comments 2 reactions 1 assignee Claimed by @gevtushenko View on GitHub
thrust
Dominant language
C++
Stars
2.5k
Forks
487
Avg merge
2d 7h
Merged PRs (30d)
296

Description

I was trying to use a single `universal_vector` to replace a pair of `host_vector` and `device_vector`, hoping to reduce memory usage and support computation with buffer size larger than GPU memory. However, it seems that `universal_vector` is very slow for `push_back` operations, regardless if the operations requires reallocation or not.

Simple benchmark:
```c++
#include
#include
#include
#include
#include

constexpr int repeat = 1000000;

int universal(bool reserve) {
thrust::universal_vector test_vector;
if (reserve) {
test_vector.reserve(repeat);
/* cudaMemPrefetchAsync(test_vector.data().get(), repeat * sizeof(int), 0, 0); */
}
std::chrono::time_point t0 = std::chrono::high_resolution_clock::now();
{
for (int i = 0; i < repeat; i++) {
test_vector.push_back(i);
}
}
std::chrono::time_point t1 = std::chrono::high_resolution_clock::now();

return std::chrono::duration_cast(t1 - t0).count();
}

int universal_set() {
thrust::universal_vector test_vector(repeat);
std::chrono::time_point t0 = std::chrono::high_resolution_clock::now();
{
for (int i = 0; i < repeat; i++) {
test_vector[i] = i;
}
}
std::chrono::time_point t1 = std::chrono::high_resolution_clock::now();

return std::chrono::duration_cast(t1 - t0).count();
}

int host_device() {
thrust::host_vector test_vector_h;
std::chrono::time_point t0 = std::chrono::high_resolution_clock::now();
for (int i = 0; i < repeat; i++) {
test_vector_h.push_back(i);
}
thrust::device_vector test_vector_d(test_vector_h);
std::chrono::time_point t1 = std::chrono::high_resolution_clock::now();
return std::chrono::duration_cast(t1 - t0).count();
}

int main() {
// warm up
for (int i = 0; i < 5; i++)
host_device();
std::cout << "host device (no reserve):" << host_device() << "ms" << std::endl;
std::cout << "universal (no reserve):" << universal(false) << "ms" << std::endl;
std::cout << "universal (with reserve):" << universal(true) << "ms" << std::endl;
std::cout << "universal (set elements):" << universal_set() << "ms" << std::endl;
std::cout << std::endl;
}
```

Output:
```
host device (no reserve):6ms
universal (no reserve):5578ms
universal (with reserve):5584ms
universal (set elements):1ms
```

I tried running `nvprof`, and got the following result:
```
==119609== NVPROF is profiling process 119609, command: ./a.out
universal (with reserve):10528ms

==119609== Profiling application: ./a.out
==119609== Profiling result:
Type Time(%) Time Calls Avg Min Max Name
GPU activities: 100.00% 1.66960s 1000000 1.6690us 1.5990us 167.81us void thrust::cuda_cub::core::_kernel_agent, int>, unsigned long>, thrust::cuda_cub::__uninitialized_fill::functor, int>, unsigned long>(thrust::cuda_cub::tag, int&)
API calls: 46.65% 4.16090s 2000000 2.0800us 573ns 269.30us cudaStreamSynchronize
30.47% 2.71802s 1000000 2.7180us 2.2110us 6.2884ms cudaLaunchKernel
10.97% 978.84ms 10000003 97ns 86ns 279.10us cudaGetLastError
5.09% 454.43ms 2000001 227ns 197ns 243.55us cudaGetDevice
2.55% 227.76ms 1000000 227ns 196ns 277.03us cudaDeviceGetAttribute
2.44% 217.78ms 2000000 108ns 87ns 29.846us cudaPeekAtLastError
1.81% 161.54ms 1 161.54ms 161.54ms 161.54ms cudaMallocManaged
0.00% 194.83us 101 1.9290us 99ns 139.97us cuDeviceGetAttribute
0.00% 156.64us 1 156.64us 156.64us 156.64us cudaFree
0.00% 17.573us 1 17.573us 17.573us 17.573us cuDeviceGetName
0.00% 10.741us 1 10.741us 10.741us 10.741us cudaFuncGetAttributes
0.00% 7.0360us 1 7.0360us 7.0360us 7.0360us cuDeviceGetPCIBusId
0.00% 1.2080us 3 402ns 150ns 887ns cuDeviceGetCount
0.00% 602ns 2 301ns 96ns 506ns cuDeviceGet
0.00% 264ns 1 264ns 264ns 264ns cuDeviceTotalMem
0.00% 245ns 1 245ns 245ns 245ns cudaGetDeviceCount
0.00% 222ns 1 222ns 222ns 222ns cuDeviceGetUuid

==119609== Unified Memory profiling result:
Device "NVIDIA GeForce GTX 1050 (0)"
Count Avg Size Min Size Max Size Total Size Total Time Name
12 - - - - 998.7450us Gpu page fault groups
```

So it seems to me that *each* `push_back` requires a `cudaStreamSynchronize`? I guess this might cause the problem, but I'm not familiar with CUDA so this might be wrong. I'm using a Geforce GTX1050, not sure if this is related to [demand-paging](https://stackoverflow.com/questions/39782746/why-is-nvidia-pascal-gpus-slow-on-running-cuda-kernels-when-using-cudamallocmana/40011988#40011988).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.