Freezing when training NeuralNetworkGPU (async).

So, I have a server app with NeuralNetworkGPU on a board.

Clients can connect and to train some data (doesn't matter).
When the second client starts training, the server freezes/stuck for a few seconds before actual training is runned (i mean before the callbacks passed to trainAsync is being called). After that few seconds I can see callbacks and the server is "unfrozen".
This happens only if GPU is used.

So, how to fix this for GPU training?
Will the more CPU cores speed up the training if I won’t use GPU?