OpenMathLib / OpenMathLib/OpenBLAS
gemm execution time is unstable in multi process system
Nessuno ha ancora preso questa issue.
- Lingua principale
- C
- Stelle
- 7.6k
- Fork
- 1.7k
- Merge medio
- 1g 3h
- PR unite (30g)
- 42
Descrizione
Hello,
We are using NVIDIA Jetson Orin platform and in a multi-process system, each process is assigned to a specific cpu and parallelizes cblas_sgem() sequentially through the remaining cores.
When the calculation is performed with only one process created, the execution time of cblas_sgem() is almost constant, but when the process increases, the calculation time becomes unstable.
Although cblas_sgem() using only one thread took about 25 ms, parallel processing with 6 threads each in 6 processes causes instability in running time from 10 ms to 60 ms.
A detailed description of the situation
We have 11 cpu cores from no. 1 to no. 11.
Create six processes, allocate them from cpu 1 to cpu 6, and synchronize the operation start time and execute them simultaneously.
Locked through Semaphore in the middle and sequentially parallelized cblas_sgem() with a total of 6 threads using 5 remaining cores from cpu 7 to cpu 11.
In this case, cblas_sgem() executed in the process allocated to cores 1 to 5 is about 11 ms, and the execution time is constant.
However, in the last process assigned to core 6, cblas_sgem() lags unsteadily from 30 ms to 60 ms, slower than when running with one thread.
The same code is executed for cores 1 to 6.
The code currently in use is as follows.
openblas_thread = 6;
openblas_set_num_threads (openblas_thread); // Set the number of openblas threads to 6
CPU_ZERO(&cpuset);
CPU_SET(sched_getcpu(), &cpuset);
pthread_setaffinity_np (pthread_self(), sizeof(cpuset), &cpuset); // reallocate itself to the original assigned CPU number (1 to 6, respectively)
for (int k = 0; k < openblas_thread-1; k++) { // allocate openblas_thread to CPU cores (7 to 11), excluding the main process(thread)
CPU_ZERO(&cpuset);
CPU_SET(11 - k, &cpuset);
openblas_setaffinity(k, sizeof(cpuset), &cpuset);
}
for(int i = 0; i < 30; i++) {
gemm_wrapper_func();
}
When I measured the execution time inside the openblas code, it was confirmed that the calculation time became unstable when processing cblas_sgem() called in the wrapper function.
Is the code above correct to use openblas normally?
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Direzione di ricerca
Iniziate dal punto di ingresso cblas_sgem() segnalato e dalle chiamate a openblas_set_num_threads() e openblas_setaffinity() nell’esempio. Riproducete il carico di lavoro con sei processi e sei thread sulla configurazione NVIDIA Jetson Orin, confrontate la variazione dei tempi con la configurazione di affinità e documentate se l’utilizzo è supportato e cosa spiega l’instabilità.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- c
- Ambito
- performance
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 30/100