[FR] Add `--offline`

There are some inconvenient workarounds for using this software without making an internet connection (which adds considerable latency on unstable networks). This use case should see official support. I propose adding the latest versions of the tokenizers to the pip package, and just using them without checking for updates when the user supplies --offline. Of course, the tokenizers can be updated any time the user doesn't use this flag.

To summarize, I propose two changes:

• Include the needed tokenizers in the initial download, such that after installing the package with pip, it is ready to go.
• Auto-update these tokenizers whenever the user does NOT supply --offline.

PS: I have skimmed the workaround, and while it works for offline usage, I am not sure it would solve the latency issue on an online machine. The workaround needs lots of manual steps, too, and it's not just some script we can run and be done with it.

Related:

• https://github.com/openai/tiktoken/issues/279
• https://github.com/openai/tiktoken/issues/232

Start by tracing how the pip package bundles tokenizers and how command-line options are handled. Define the update behavior for normal use versus --offline, then verify that a fresh pip installation works without network access and that offline use avoids update checks.