Per device memory allocation

Hi,

I have a multi-device network with custom nodes in the CG. As of now my nodes use a memory allocator I made, but I want to move the one of dynet (i.e. use aux_memory). They are based on dynamic programming so they need a lot of memory, which is not alway the case of the rest of the network that runs on GPU.

Dynet provides a command line option for memory allocation but it cannot be specified per device: `--dynet-mem FOR,BACK,PARAM,SCRATCH`.
Is there any reason for that?

Best,
Caio

Start by tracing how the `--dynet-mem FOR,BACK,PARAM,SCRATCH` option is handled and how `aux_memory` is used for custom nodes. Determine whether memory settings can be scoped to individual devices and clarify the expected behavior for multi-device networks. Done should mean a defined, tested per-device allocation mechanism, but the issue does not name relevant files or tests.

Written by the indexing model from the issue text.