allenai / allenai/allenact

Testing consistency for distributed training

未关闭
#228 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Python
星标
383
派生
59
PR 合并指标
30 天内没有已合并 PR

描述

## Problem

It is challenging to test that training is perfectly consistent when doing distributed training. I.e. if I change the number of GPUs I'm training with but keep the number of processes the same, do my models I get exactly the same gradients?

## Desired solution

We should create an experiment config in AI2-THOR that is seeded such that the agents will see the same scenes and take the same actions regardless of the number of GPUs. In this case, we can test that the gradients are exactly the same in different GPU configurations.

## Additional context

Issue based on the observation by @marlohmann that training curves may be different based on the number GPUs used.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。