Troubleshooting A3C agent

Hello,
as a part of my master thesis, I have been trying to reproduce the results from the DeepMind paper. I have implemented the A3C algorithm and am currently testing it on the MoveToBeacon minigame. However, I am having troubles with learning a good policy. None of my runs have exceeded average reward of 2. However, I have pretty limiting hardware; I am not able to run more than 3600 simulation steps per minute (including the loss function minimisation and having multiple simulations running at the same time, 1 step = 8 actual game steps). So it takes nearly 5 hours of training to get to 1 million simulation steps.

However, I am not sure what I am doing wrong. It is possible that there is somewhere some mistake in my implementation (it should be in accordance with the A3C original paper). It is possible that I have set wrong hyperparameters. It is possible that the learning is so slow that it seems that no learning happens (I usually don't let it run much past 1 million steps). But I can't tell with certainty which one it is until I successfully train an agent.

In the paper, you mention some of the hyper-parameter settings (learning rate from uniform(1e-3,1e-5), entropy loss coef 1e-3, 40 steps unroll of BPTT, 64 threads).

However, you don't mention some of the others, specifically, from what distribution have you sampled the value gradient coefficient and the T_max of A3C (meaning the max number of steps for computing the n-step reward)? How important is the number of parallel agents training? To me, there seems to be little difference in training with 10 agents and with 32 agents.

Also, what would be your recommendation on determining where is the problem with my implementation? How fast should the correct algorithm be expected to find a reasonable policy on the MoveToBeacon minigame? In another issue, someone said that after couple tens of thousands, it should already be good.

Any insight greatly appreciated!