tiktoken under counting tokens for openai text-embedding-3-large

Starting on January 2nd 2025 we started noticing errors in our logs that we were over the context limit when creating text-embedding-3-large embeddings on openai. I believe there may have been a change on the openai side since we hadn't made any related changes. In one case I looked into tiktoken reported 7995 tokens but we received the following error from openai:

{'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 8781 tokens (8781 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}

For the moment we're reducing our limit for max tokens to mitigate, but wanted to raise the issue.

Start by reproducing the reported discrepancy for text-embedding-3-large, comparing the token count from tiktoken with the token count rejected by OpenAI. No file or test is named in the issue, so locate the model encoding configuration and existing token-count tests first. Done means identifying the cause and adding a regression test that prevents under-counting.