allenai / allenai/allenact

Testing consistency for distributed training

Open
#228 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
383
Forks
59
PR merge metrics
No merged PRs in 30d

Description

## Problem

It is challenging to test that training is perfectly consistent when doing distributed training. I.e. if I change the number of GPUs I'm training with but keep the number of processes the same, do my models I get exactly the same gradients?

## Desired solution

We should create an experiment config in AI2-THOR that is seeded such that the agents will see the same scenes and take the same actions regardless of the number of GPUs. In this case, we can test that the gradients are exactly the same in different GPU configurations.

## Additional context

Issue based on the observation by @marlohmann that training curves may be different based on the number GPUs used.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.