Deleting Conda/Python as a dependency entirely to dramatically decrease "latency to step"

Following up on this [tweet](https://x.com/karpathy/status/1795501945832247790), copy pasting, and just creating an Issue as a TODO.

"""
The thing that makes this a bit complicated right now is the start latency. What bloats up the setup time right now is the dataset and its tokenization, which is all done in Python right now. Installing huggingface datasets, downloading FineWeb 10B and tokenizing it is currently ~1 hr. I think I have to look into precomputing all of this and just saving the final .bin files (20GB) of tokens somewhere (S3 or so?). You could imagine fetching data shards asynchronously while the training started. This would completely eliminate any Python dependency.

The next slightly annoying thing is cuDNN, which is a 2GB download and installation, just to get the flash attention kernel. And it compiles for 1.5 minutes. But NVIDIA reached out and mentioned they are trying to bring this down a lot.

In principle, the code should compile and run roughly instantaneously.
"""

TLDR I think I'll pre-tokenize FineWeb100B with GPT-2 tokenizer, zip up the .bin shards, and put them up somewhere (e.g. S3?). And then we could just download, unzip, and directly train without any Python involvement at all.

TODO think through a bit.

No files, tests, or entry points are named. Start by tracing the current Python-based FineWeb download and tokenization setup, then assess pre-tokenized GPT-2 .bin shards, optional S3 storage, and asynchronous fetching. Done means training can start without the Conda/Python dependency while addressing the cuDNN setup latency described here.

Written by the indexing model from the issue text.