non-deterministic behavior on the GPU?

I am unsure why, but we observe some non-deterministic behavior when running on the GPU.
The dynet-seed is set (as well as the python seed and the numpy seed), yet the losses in different runs start to diverge after a while (by the 100th update, is almost surely happens).

This does not happen with the same code on the CPU. Maybe there is a race condition somewhere?

See the following gist:
https://gist.github.com/yoavg/510154a959e76370627a81da173cec9b

`python not_deterministic.py > t1`
`python not_deterministic.py > t2`
`diff t1 t1`