google-deepmind / google-deepmind/mujoco_playground

NAN from the simulation will ruin the training

Open
#240 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.2k
Forks
359
Avg merge
7d 3h
Merged PRs (30d)
1

Description

https://github.com/google-deepmind/mujoco_playground/blob/bfac26213883acd08a104fd614a23021b251966d/mujoco_playground/_src/dm_control_suite/humanoid.py#L119
When training with the DM_Control environment, it will face nan problem because of the simulation instability. The previous code show that we will set the environment to done if nan is detected. However we didn't do this operation to the reward and I think reward will pass to the loss computation and thus pollute the critic, policy and other value relevant to reward. As observed in my training, after the nan appear then all the reward will be nan and training loss will also be nan.
The solution of this problem is to add the same detection as done. If the nan is detected then set the reward to zero to cut the spread of nan.
```
def step(self, state: mjx_env.State, action: jax.Array) -> mjx_env.State:
data = mjx_env.step(self.mjx_model, state.data, action, self.n_substeps)
reward = self._get_reward(data, action, state.info, state.metrics) # pylint: disable=redefined-outer-name
obs = self._get_obs(data, state.info)
done = jp.isnan(data.qpos).any() | jp.isnan(data.qvel).any()
done = done.astype(float)
**reward = (1-done) * reward**
return mjx_env.State(data, obs, reward, done, state.metrics, state.info)
```
Training instability is a common problem which I think not only in JAX based MJX but also in CPU based MuJoCo. Is this modification rational?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.