Context size & token usage calculation seems to be incorrect

With a simple "Hi" message sent as the very first message and without any other active chats, the token usage shows 9.2K tokens used.

Also, when set the context to 4K from the existing 65536, I get an error

This is my entire conversation history:

_hi
Hello! How can I help you today?

why the token usage shows 9.2k tokens for a simple hi message ?
The 9.2k token usage for a simple "hi" message is likely due to:

Context overhead: The system includes metadata, conversation history, and internal processing that counts toward token usage, even for simple greetings.

Model-specific counting: Different tokenization methods count tokens differently. Some models count every character or word, while others use subword tokenization (like BPE - Byte Pair Encoding).

System prompts and instructions: The system prompt and any default instructions you've set up are included in the token count.

API overhead: Some APIs add overhead for authentication, routing, and other infrastructure costs.

Batch processing: If you're using batch inference or have context from previous conversations, those tokens are included.

The exact breakdown would depend on your specific model configuration and API settings. Would you like me to help you optimize token usage or understand your specific tokenization setup?_

For this history, the total token usage is 18.1K

No source files, tests, or entry points are named in the report. Reproduce the issue with a first-message "Hi" and with context reduced from 65536 to 4K, then trace the token-usage calculation and context-limit error; done means the displayed usage matches the conversation and a 4K context setting works without the reported error.

Written by the indexing model from the issue text.