Different Values on CPU vs GPU.

[I was creating a NER tagger on Conll 2003 data based on the Ma and Hovy tagger.](https://github.com/blester125/dynet/blob/competative-tagger/examples/tagger/conv_blstm_crf.py)

I was getting really poor performance.
```
[tagger] Best f1 was 0.908426 at epoch 14
[tagger] Stopping at epoch 55 due to failure to improve.
[tagger] Test F1: 0.853864
```
This poor performance was both at the end of training and after the first epoch I was getting dev F1 around `78` compared to the F1 of around `85` I normally see in a similar tagger I have written in pytorch.

I then ran the tagger on the gpu and suddenly I getting much better performance.
```
[tagger] Best f1 was 0.934888 at epoch 36
[tagger] Stopping at epoch 77 due to failure to improve.
[tagger] Test F1: 0.890261
```
```
[tagger] Best f1 was 0.935584 at epoch 15
[tagger] Stopping at epoch 56 due to failure to improve.
[tagger] Test F1: 0.895778
```
I was also seeing F1 scores after the first epoch that were comparable to the scores from the the pytorch tagger.

I investigated this a bit more and it seems that running things on CPU vs GPU produce very different results. I created a [script here](https://github.com/blester125/dynet/blob/competative-tagger/examples/tagger/dynet_cpu_gpu.py) a simple convolution and tries to optimize the result to zero.

Running it you can see that the final values of parameters are very different:

* Parts of the embeddings layer
```
CPU:
[ 0.09190765 -0.35686716 -0.08511744 0.62517864 0.27539271]
GPU:
[ 0.0408158 -0.3782683 -0.09938584 0.61804533 0.38606855]
```
* Parts of the Conv weights
```
CPU:
[[[[-0.00934861 -0.45052513 0.38271371 0.37630072 -0.23521839
0.05024116 0.03176426 0.07979026 0.42238858 -0.27316907]
GPU:
[[[[ 0.0184729 -0.422703 0.41053453 0.40412155 -0.20739615
0.0780626 0.05958565 0.10761151 0.45020884 -0.24534746]
```
* Parts of the Conv Bias
```
CPU:
[-0.34024909 0.05928647 -0.36520264 -0.31748942 0.42725024 0.25176921
0.38832936 0.32116064 -0.14631058 -0.34010348]
GPU:
[-0.33341098 0.06612484 -0.35836464 -0.31065124 0.43408799 0.25860667
0.39516699 0.32799882 -0.13947366 -0.33326536]
```

Not only are the final weights different but the final result is very different with the GPU getting much closer to the goal than the CPU version:
```
CPU Loss: [0.85731989]
GPU Loss: [0.28159717]
```

I am using the `MomentumSGDTrainer` and I thought at first it might have something to do with the sparse updates because it mentioned they are not strictly correct for Momentum but even turning them off still has this error. When I run this script with just the `SimpleSGDTrainer` this discrepancy between GPU and CPU goes away.

Start by running examples/tagger/dynet_cpu_gpu.py and compare its CPU and GPU results with MomentumSGDTrainer and SimpleSGDTrainer. Read the linked examples/tagger/conv_blstm_crf.py for the larger NER symptom, then trace the trainer behavior that differs by device. Done means CPU and GPU produce consistent optimization results, with a regression test for the discrepancy.

Written by the indexing model from the issue text.