Tokenizer boundary impact

Overview

The client and the server interact by sending text to each other. Specifically, the client sends a prompt to the server which tokenizes it, and then return the response in text. The response from the server can be a single chunk (non-streaming) or can be divided into multiple chunks (streaming). In case of streaming response, the client will receive a stream of chunks of text and aggregate them to run the tokenizer at the end and count the tokens. This is used to derive metrics such as TPOT and TPS. TTFT is computed by just observing when the first chunk arrives - so it is less impacted.
The concern is that the chunk boundary can lead to different token counts. For instance, with o200k_harmony tokenizer:

' Infrastructure' — rank 70223

' uncertainties' — rank 97628

' information' — rank 2164

'function' — rank 2706

' something' — rank 3543

' available' — rank 2839

Full split table for 'function'

'f'       | 'unction'   -> ['f']              + ['unction']          = 2 tokens
'fu'      | 'nction'    -> ['fu']             + ['n', 'ction']       = 3 tokens
'fun'     | 'ction'     -> ['fun']            + ['ction']            = 2 tokens
'func'    | 'tion'      -> ['func']           + ['tion']             = 2 tokens
'funct'   | 'ion'       -> ['fun', 'ct']      + ['ion']              = 3 tokens
'functi'  | 'on'        -> ['fun', 'ct', 'i'] + ['on']               = 4 tokens  ← worst
'functio' | 'n'         -> ['fun', 'ct', 'io']+ ['n']                = 4 tokens

Challenge

The token count will be generated by looking at individual chunks returned by the server over OpenAI API completion protocol. This might not be the exact same number of tokens that are generated by the server, but in the absence of any viable alternative, is the best course of action.

The issue names no files, tests, or implementation entry point; begin by tracing token counting for streamed responses received through the OpenAI API completion protocol. Done is not defined: clarify the intended counting behavior, the acceptable boundary error, and how it should be verified against whole-response tokenization.