ProcessExitedException: process 0 (2x 4090)

I'm trying what looks like the "Hello World" of this repo: Running the basic training on a Runpod community cloud `2 x RTX 4090, (128 vCPU 125 GB RAM)` configuration. Normally I'd play around with this for longer before posting an issue, but since Runpod was mentioned explicitly in the Answer.ai intro post, I figure this will be the simplest path for anybody trying to test this out.

On their `runpod/pytorch:2.2.0-py3.10-cuda12.1.1-devel-ubuntu22.04` pod:
```
python train.py \
--model_name meta-llama/Llama-2-70b-hf \
--batch_size 2 \
--context_length 2048 \
--precision bf16 \
--train_type qlora \
--use_gradient_checkpointing true \
--use_cpu_offload true \
--dataset alpaca \
--reentrant_checkpointing true \
--log_to wandb
```

Download the Llama-2 mode, sets everything up, and dies with the following backtrace:

```
Traceback (most recent call last):
File "/root/fsdp_qlora/train.py", line 939, in
def main(
File "/usr/local/lib/python3.10/dist-packages/fastcore/script.py", line 125, in call_parse
return _f()
File "/usr/local/lib/python3.10/dist-packages/fastcore/script.py", line 119, in _f
return tfunc(**merge(args, args_from_prog(func, xtra)))
File "/root/fsdp_qlora/train.py", line 1010, in main
mp.spawn(fsdp_main,
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 241, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 197, in start_processes
while not context.join():
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 140, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGKILL
```

Log:
```
1 Creating model 0
2 Loading model 0
3 Model created 0 1.119 GB
4 trainable params: 744,488,960 || all params: 35,495,616,512 || trainable%: 2.097410985236193
5 Wrapping model w/ FSDP 0
6 Wrapped model 0 1.444 GB
7 Applying activation checkpointing 0
8 Total Training Steps: 12940
9 Epoch 0, Loss 0.000: 0%| | 0/12940 [00:00

Start by reproducing the command on the specified Runpod image, then inspect train.py around fsdp_main and the mp.spawn call near line 1010. Use the SIGKILL timing, logs, and linked W&B run to determine the failure cause; done means the two-GPU training command runs past initialization or the resource limitation and required change are documented.

Written by the indexing model from the issue text.