deepseek-ai / deepseek-ai/DeepSeek-OCR

Cant able to run finetuned deepseek ocr on vllm, Help please!

Open
#311 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
23.9k
Forks
2.2k
PR merge metrics
No merged PRs in 30d

Description

```

# Install vLLM from nightly build
!pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
from vllm import LLM, SamplingParams
from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor
from PIL import Image
# Create model instance
llm = LLM(
model="unsloth/DeepSeek-OCR",
enable_prefix_caching=False,
mm_processor_cache_gb=0,
logits_processors=[NGramPerReqLogitsProcessor]
)

```
Base model works but when i use finetuned model

```
# Create model instance
llm = LLM(
model="/content/drive/MyDrive/unsloth_finetune",
enable_prefix_caching=False,
mm_processor_cache_gb=0,
logits_processors=[NGramPerReqLogitsProcessor]
)
4 frames
[/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py](https://localhost:8080/#) in __init__(__dataclass_self__, *args, **kwargs)
119 __tracebackhide__ = True
120 s = __dataclass_self__
--> 121 s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
122
123 __init__.__qualname__ = f'{cls.__qualname__}.__init__'

ValidationError: 1 validation error for ModelConfig
Value error, Invalid repository ID or local directory specified: '/content/drive/MyDrive/unsloth_finetune'.
Please verify the following requirements:
1. Provide a valid Hugging Face repository ID.
2. Specify a local directory that contains a recognized configuration file.
- For Hugging Face models: ensure the presence of a 'config.json'.
- For Mistral models: ensure the presence of a 'params.json'.
[type=value_error, input_value=ArgsKwargs((), {'model': ...rocessor_plugin': None}), input_type=ArgsKwargs]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.