arrayfire / arrayfire/arrayfire

[BUG] device memory leak in cuBLAS (matmul)

オープン
#3,286 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
bug
主要言語
C++
スター
4.9k
フォーク
555
平均マージ
1時間 24分
マージ済み PR(30日)
1

説明

Each launch of a new threat leaves around 10KB of device memory allocated.

Description
===========
Launching a function with a matmul operation in a loop of consecutive threads on CUDA will result in memory overflow on the device, and exception: CUBLAS Error (3): CUBLAS_STATUS_ALLOC_FAILED. Running the same function in a loop on the main thread operates normally, even after 10,000 loops.
On OpenCL, the thread version as the main thread version operate normally.
Full trace logging, indicates the existing device buffers are reused as expected (also in the separate threads).

Arrayfire build: master 3.9.0 b05da694
Back-end: CUDA
Workaround available: No
Reproducibility: Yes
Logging: [Logging.txt](https://github.com/arrayfire/arrayfire/files/9490460/Logging.txt)
Device memory in main thread: ![device memory main thread](https://user-images.githubusercontent.com/70607676/188469612-e97a9422-551a-4255-8f7d-a7e689a1b349.png)

Device memory in consecutive threads: ![device memory threads](https://user-images.githubusercontent.com/70607676/188469589-b3cc65e0-0864-44b5-9bcc-6dc27aa639d9.png)

Reproducible Code and/or Steps
------------------------------
```
int main() {
class trainer {
int device;

public:
trainer(const int device) : device(device){};
void train() {
af::setDevice(device);
const af::array a{af::iota(af::dim4(10, 10))};
// size has no impact
af::array c{af::matmul(a, a)};
};
};

try {
af::info();
trainer trainers{af::getDevice()};

for (int i{0}; i < 1000; ++i) {
std::cout << i << ", ";
// OK
trainers.train();

// trows exception
// std::thread t(&trainer::train, std::ref(trainers));
// t.join();
}
} catch (af::exception &ae) { std::cerr << ae.what() << std::endl; }
return 0;
}
```

System Information
------------------
1. ArrayFire version : master 3.9.0 b05da694
2. Devices installed on the system: GTX 750 Ti
3. (optional) Output from the af::info() function if applicable: see logging
4. Output from the following scripts:
[Output cmds.txt](https://github.com/arrayfire/arrayfire/files/9490479/Output.cmds.txt)

Checklist
---------

- [x] Using the latest available ArrayFire release
- [x] GPU drivers are up to date

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start by running the provided C++ reproducer with the CUDA backend, comparing af::matmul in the main thread with consecutive std::thread launches, and review Logging.txt. Trace device allocation and cleanup around af::setDevice and matmul; done means repeated threaded launches no longer consume device memory or end with CUBLAS_STATUS_ALLOC_FAILED.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
cpp
領域
hpc, performance
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。