"cat_cuda" not implemented for 'Float8_e4m3fn'

torch 2.4.0
flux1-dev-fp8-e4m3fn.safetensors
t5xxl_fp8_e4m3fn.safetensors

Settings:
--fp8_base
--split_mode

Error:
[rank0]: RuntimeError: "cat_cuda" not implemented for 'Float8_e4m3fn'

PyTorch doesn't yet support Float8_e4m3fn for torch.cat (probably), but "fp8_base" should be able to handle float8_e4m3fn models.

```
[rank0]: RuntimeError: "cat_cuda" not implemented for 'Float8_e4m3fn'
[rank0]:[W1018 14:07:24.358993691 ProcessGroupNCCL.cpp:1168] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
W1018 14:07:27.587000 138783073695552 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 136 closing signal SIGTERM
E1018 14:07:27.619000 138783073695552 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 135) of binary: /opt/conda/bin/python3.10
Traceback (most recent call last):
File "/opt/conda/bin/accelerate", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1097, in launch_command
multi_gpu_launcher(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 734, in multi_gpu_launcher
distrib_run.run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
flux_train_network.py FAILED
------------------------------------------------------------
Failures:

------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-10-18_14:07:27
host : 70d34f90e91b
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 135)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
add Codeadd Markdown
```

Start at the flux_train_network.py entry point and trace handling for the --fp8_base and --split_mode settings with the listed Float8_e4m3fn models. Reproduce the failure under PyTorch 2.4.0 and verify that the training run no longer raises the cat_cuda error.

Written by the indexing model from the issue text.