Lockless support for parallelism?

Hello,

I am the studying the support for parallelism on Dynet and I've come across this description:

> Internally, DyNet launches a pool of training processes and automatically handles passing data examples to each worker. Each worker process individually processes a datum, computing the results of the forward and backward passes, computes gradients with respect to each parameter, and passes these results back to the parent process via a shared memory variable. **Whenever the parent process, which is also processing data, completes a gradient computation, it averages all of the gradients currently in the shared memory gradient storage and updates all parameters with respect to that average gradient**. In this way running training on n cores is similar to training with a stochastic minibatch size with expected value of approximately n. This method is quite efficient, achieving nearly linear speedups with increasing numbers of cores, **due to its lockless nature**.

from here https://github.com/clab/dynet/blob/master/doc/source/multiprocessing.rst

Is this still true?

Then, I found this snippet of code from https://github.com/clab/dynet/blob/v2.0/dynet/mp.h#L252:

bool do_update = !header.is_dev_set && cid == 0;
unsigned counter = 0;
if (!header.is_dev_set) {
shared_object->counter_mutex.wait();
counter = ++shared_object->counter;
if (do_update) { shared_object->counter = 0; }
shared_object->counter_mutex.post();
}
if (do_update && trainer != nullptr) {
shared_object->update_mutex.wait();
trainer->update(1.0 / counter);
shared_object->update_mutex.post();
}
if (batch_counter == header.report_frequency) {
if (cid == 0) {
std::cerr << (header.is_dev_set ? "dev" : "train") << " loss: " << batch_loss << std::endl;
}
batch_loss = S();
batch_counter = 0;
}
}
if (header.end_of_epoch && trainer != nullptr) {
trainer->update_epoch();
}

// Let the parent know that we're done and return the loss value
write_data(workloads[cid].c2p[1], total_loss);

From my understanding, the _update_mutex_ is acquired before the gradients are updated. So in fact, locks are used?

Also, could you please tell me in which part of the code this is performed by the parent process?

_it averages all of the gradients currently in the shared memory gradient storage and updates all parameters with respect to that average gradient_

I could found how you pass the losses from the children to the parents, but not the aforementioned averaging of gradients.

Thanks in advance.

Start with doc/source/multiprocessing.rst and dynet/mp.h around the cited v2.0 code. Trace the counter_mutex, update_mutex, shared gradient storage, and parent/worker paths to determine whether the lockless description is still accurate and where averaging occurs. Done means the documentation answers both questions and identifies the relevant implementation path.

Written by the indexing model from the issue text.