AttributeError: 'Encoding' object has no attribute 'num_tokens'

Nobody has claimed this yet.

fschat==0.2.36

```
WARNING 02-29 16:06:12 config.py:140] gptq quantization is not fully optimized yet. The speed can be slower than non-quantized models.
INFO 02-29 16:06:12 llm_engine.py:72] Initializing an LLM engine with config: model='/mnt/Qwen-1_8B-Chat-Int4', tokenizer='/mnt/Qwen-1_8B-Chat-Int4', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=gptq, seed=0)
WARNING 02-29 16:06:13 tokenizer.py:66] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
2024-02-29 16:06:15 | INFO | datasets | PyTorch version 2.2.0 available.
INFO 02-29 16:06:25 llm_engine.py:219] # GPU blocks: 4060, # CPU blocks: 1365
2024-02-29 16:06:32 | INFO | model_worker | Loading the model ['Qwen-1_8B-Chat-Int4'] on worker 92230372, worker type: vLLM worker...
2024-02-29 16:06:32 | INFO | model_worker | Register to controller
2024-02-29 16:06:32 | ERROR | stderr | INFO: Started server process [65]
2024-02-29 16:06:32 | ERROR | stderr | INFO: Waiting for application startup.
2024-02-29 16:06:32 | ERROR | stderr | INFO: Application startup complete.
2024-02-29 16:06:32 | ERROR | stderr | INFO: Uvicorn running on http://0.0.0.0:21002 (Press CTRL+C to quit)
2024-02-29 16:07:17 | INFO | model_worker | Send heart beat. Models: ['Qwen-1_8B-Chat-Int4']. Semaphore: None. call_ct: 0. worker_id: 92230372.
2024-02-29 16:08:02 | INFO | model_worker | Send heart beat. Models: ['Qwen-1_8B-Chat-Int4']. Semaphore: None. call_ct: 0. worker_id: 92230372.
2024-02-29 16:08:08 | INFO | stdout | INFO: 127.0.0.1:60300 - "POST /worker_get_conv_template HTTP/1.1" 200 OK
2024-02-29 16:08:08 | INFO | stdout | INFO: 127.0.0.1:60308 - "POST /model_details HTTP/1.1" 200 OK
2024-02-29 16:08:08 | INFO | stdout | INFO: 127.0.0.1:60320 - "POST /count_token HTTP/1.1" 500 Internal Server Error
2024-02-29 16:08:08 | ERROR | stderr | ERROR: Exception in ASGI application
2024-02-29 16:08:08 | ERROR | stderr | Traceback (most recent call last):
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/fastchat/serve/base_model_worker.py", line 156, in count_token
2024-02-29 16:08:08 | ERROR | stderr | input_ids = self.tokenizer(prompt).input_ids
2024-02-29 16:08:08 | ERROR | stderr | TypeError: 'Encoding' object is not callable
2024-02-29 16:08:08 | ERROR | stderr |
2024-02-29 16:08:08 | ERROR | stderr | During handling of the above exception, another exception occurred:
2024-02-29 16:08:08 | ERROR | stderr |
2024-02-29 16:08:08 | ERROR | stderr | Traceback (most recent call last):
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
2024-02-29 16:08:08 | ERROR | stderr | result = await app( # type: ignore[func-returns-value]
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
2024-02-29 16:08:08 | ERROR | stderr | return await self.app(scope, receive, send)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/fastapi/applications.py", line 1054, in __call__
2024-02-29 16:08:08 | ERROR | stderr | await super().__call__(scope, receive, send)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/applications.py", line 123, in __call__
2024-02-29 16:08:08 | ERROR | stderr | await self.middleware_stack(scope, receive, send)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/middleware/errors.py", line 186, in __call__
2024-02-29 16:08:08 | ERROR | stderr | raise exc
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/middleware/errors.py", line 164, in __call__
2024-02-29 16:08:08 | ERROR | stderr | await self.app(scope, receive, _send)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
2024-02-29 16:08:08 | ERROR | stderr | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-02-29 16:08:08 | ERROR | stderr | raise exc
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-02-29 16:08:08 | ERROR | stderr | await app(scope, receive, sender)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/routing.py", line 758, in __call__
2024-02-29 16:08:08 | ERROR | stderr | await self.middleware_stack(scope, receive, send)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/routing.py", line 778, in app
2024-02-29 16:08:08 | ERROR | stderr | await route.handle(scope, receive, send)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/routing.py", line 299, in handle
2024-02-29 16:08:08 | ERROR | stderr | await self.app(scope, receive, send)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/routing.py", line 79, in app
2024-02-29 16:08:08 | ERROR | stderr | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-02-29 16:08:08 | ERROR | stderr | raise exc
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-02-29 16:08:08 | ERROR | stderr | await app(scope, receive, sender)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/starlette/routing.py", line 74, in app
2024-02-29 16:08:08 | ERROR | stderr | response = await func(request)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/fastapi/routing.py", line 278, in app
2024-02-29 16:08:08 | ERROR | stderr | raw_response = await run_endpoint_function(
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
2024-02-29 16:08:08 | ERROR | stderr | return await dependant.call(**values)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/fastchat/serve/vllm_worker.py", line 231, in api_count_token
2024-02-29 16:08:08 | ERROR | stderr | return worker.count_token(params)
2024-02-29 16:08:08 | ERROR | stderr | File "/root/miniconda3/lib/python3.9/site-packages/fastchat/serve/base_model_worker.py", line 159, in count_token
2024-02-29 16:08:08 | ERROR | stderr | input_echo_len = self.tokenizer.num_tokens(prompt)
2024-02-29 16:08:08 | ERROR | stderr | AttributeError: 'Encoding' object has no attribute 'num_tokens'
2024-02-29 16:08:47 | INFO | model_worker | Send heart beat. Models: ['Qwen-1_8B-Chat-Int4']. Semaphore: None. call_ct: 0. worker_id: 92230372.
```

Start in fastchat/serve/base_model_worker.py around count_token and then inspect fastchat/serve/vllm_worker.py around api_count_token. Reproduce the POST /count_token failure with the reported setup and trace the tokenizer object used there. Done means the endpoint handles this tokenizer without the shown TypeError or AttributeError.

Written by the indexing model from the issue text.