abi / abi/screenshot-to-code

Ollama Support

Offen
#354 27 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
model p1
Vorherrschende Sprache
Python
Sterne
78.7k
Forks
9.6k
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

I love your project, I want to use it with local ollama+llava and i tried many way including asking chat gpt.
I am on Windows 11, i tried docker and no go. changed api address from settings in frontend also
```
api key as "ollama" and
Api Url as http://localhost:11434/v1/
```
and i tested my local ollama+llava answering and running with postman.

changed frontend\src\lib\models.ts

```
// Keep in sync with backend (llm.py)
// Order here matches dropdown order
export enum CodeGenerationModel {
GPT_4O_2024_05_13 = "llava", --> HERE
GPT_4_TURBO_2024_04_09 = "gpt-4-turbo-2024-04-09",
GPT_4_VISION = "gpt_4_vision",
CLAUDE_3_SONNET = "claude_3_sonnet",
}

// Will generate a static error if a model in the enum above is not in the descriptions
export const CODE_GENERATION_MODEL_DESCRIPTIONS: {
[key in CodeGenerationModel]: { name: string; inBeta: boolean };
} = {
"llava": { name: "LLava", inBeta: false }, --> and here
"gpt-4-turbo-2024-04-09": { name: "GPT-4 Turbo (Apr 2024)", inBeta: false },
gpt_4_vision: { name: "GPT-4 Vision (Nov 2023)", inBeta: false },
claude_3_sonnet: { name: "Claude 3 Sonnet", inBeta: false },
};

```
also backend\llm.py

# Actual model versions that are passed to the LLMs and stored in our logs
```
class Llm(Enum):
GPT_4_VISION = "gpt-4-vision-preview"
GPT_4_TURBO_2024_04_09 = "gpt-4-turbo-2024-04-09"
GPT_4O_2024_05_13 = "llava"
CLAUDE_3_SONNET = "claude-3-sonnet-20240229"
CLAUDE_3_OPUS = "claude-3-opus-20240229"
CLAUDE_3_HAIKU = "claude-3-haiku-20240307"

Will throw errors if you send a garbage string
def convert_frontend_str_to_llm(frontend_str: str) -> Llm:
if frontend_str == "gpt_4_vision":
return Llm.GPT_4_VISION
elif frontend_str == "claude_3_sonnet":
return Llm.CLAUDE_3_SONNET
elif frontend_str == "llava":
return Llm.GPT_4O_2024_05_13
else:
return Llm(frontend_str)
```
console and backend errors below

```
INFO: ('127.0.0.1', 57364) - "WebSocket /generate-code" [accepted]
Incoming websocket connection...
INFO: connection open
Received params
Generating html_tailwind code for uploaded image using Llm.GPT_4O_2024_05_13 model...
Using OpenAI API key from client-side settings dialog
Using OpenAI Base URL from client-side settings dialog
generating code...
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\uvicorn\protocols\websockets\websockets_impl.py", line 250, in run_asgi
result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\applications.py", line 276, in __call__
await super().__call__(scope, receive, send)
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\applications.py", line 122, in __call__
await self.middleware_stack(scope, receive, send)
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\errors.py", line 149, in __call__
await self.app(scope, receive, send)
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\cors.py", line 75, in __call__
await self.app(scope, receive, send)
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
raise exc
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
raise e
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
await self.app(scope, receive, send)
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\routing.py", line 718, in __call__
await route.handle(scope, receive, send)
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\routing.py", line 341, in handle
await self.app(scope, receive, send)
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\starlette\routing.py", line 82, in app
await func(session)
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\fastapi\routing.py", line 289, in app
await dependant.call(**values)
File "C:\Users\k\screenshot-to-code\backend\routes\generate_code.py", line 262, in stream_code
completion = await stream_openai_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\k\screenshot-to-code\backend\llm.py", line 60, in stream_openai_response
stream = await client.chat.completions.create(**params) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\k\AppData\Local\pypoetry\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\openai\resources\chat\completions.py", line 1334, in create
return await self._post(
^^^^^^^^^^^^^^^^^
y\Cache\virtualenvs\backend-bYKjg4sG-py3.11\Lib\site-packages\openai\_base_client.py", line 1532, in _request
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
INFO: connection closed
```
![sstc2](https://github.com/abi/screenshot-to-code/assets/7223210/909a0a6e-421d-4114-b981-3325af328610)
![Ekran görüntüsü 2024-06-05 192114](https://github.com/abi/screenshot-to-code/assets/7223210/ccbfe31a-0366-43f2-9176-594b142db409)
![sstc](https://github.com/abi/screenshot-to-code/assets/7223210/c1bae4e4-f06a-4ba8-aa69-88bb1d7c81c4)
![Ekran görüntüsü 2024-06-05 192036](https://github.com/abi/screenshot-to-code/assets/7223210/a46f48e5-33b5-476e-8abb-c82922347408)

If can be use on local server it'll be awesome!
Thanks for consideration

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.