abi / abi/screenshot-to-code

Support multiple selection in "Select and Edit"

Offen
#418 5 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
78.7k
Forks
9.6k
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

**Describe the bug**

I am using VS code insider in admin mode.

In backend .env I entered my AI keys:
OPENAI_API_KEY=sk-2siLny...
ANTHROPIC_API_KEY=sk-ant-api0...

When drag/drop an .mp4 video below:
https://github.com/user-attachments/assets/22713a47-4d23-44e9-b83f-dcb774ebbcc8
I am getting a notification dialog:
**Error assembling prompt. Contact support at support@picoapps.xyz**

How can I use the latest models and what it the code I should change?
**I want to use the latest OppenAI model: o1-preview which points to o1-preview-2024-09-12
and
I want to use the lastes Anthropic model: claude-3-5-sonnet-latest which points to claude-3-5-sonnet-20241022**

**I am confused where in the code I can designate the latest versions**

When I drag/drop .png image:
screenshot1
I cannot see Option 1 rendering.
What is that Option 1 suppose to show? Open AI based generation?

How can I use the latest o1 model

frontend\src\lib\models.ts
`// Keep in sync with backend (llm.py)
// Order here matches dropdown order
export enum CodeGenerationModel {
CLAUDE_3_5_SONNET_2024_06_20 = "claude-3-5-sonnet-20240620",
GPT_4O_2024_05_13 = "gpt-4o-2024-05-13",
GPT_4_TURBO_2024_04_09 = "gpt-4-turbo-2024-04-09",
GPT_4_VISION = "gpt_4_vision",
CLAUDE_3_SONNET = "claude_3_sonnet",
}

// Will generate a static error if a model in the enum above is not in the descriptions
export const CODE_GENERATION_MODEL_DESCRIPTIONS: {
[key in CodeGenerationModel]: { name: string; inBeta: boolean };
} = {
"gpt-4o-2024-05-13": { name: "GPT-4o", inBeta: false },
"claude-3-5-sonnet-20240620": { name: "Claude 3.5 Sonnet", inBeta: false },
"gpt-4-turbo-2024-04-09": { name: "GPT-4 Turbo (deprecated)", inBeta: false },
gpt_4_vision: { name: "GPT-4 Vision (deprecated)", inBeta: false },
claude_3_sonnet: { name: "Claude 3 (deprecated)", inBeta: false },
};`

**Console log for backend:**

Using openAiApiKey from client-side settings dialog
Using anthropicApiKey from client-side settings dialog
Using official OpenAI URL
Generating react_tailwind code in video mode using Llm.CLAUDE_3_5_SONNET_2024_06_20...
Status (variant 0): Generating code...
Status (variant 1): Generating code...
C:\Users\Greg\AppData\Local\Temp\tmpkwxwbc8s.mp4
Error assembling prompt. Contact support at support@picoapps.xyz
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\moviepy\video\io\ffmpeg_reader.py", line 285, in ffmpeg_parse_infos
line = [l for l in lines if keyword in l][index]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\uvicorn\protocols\websockets\websockets_impl.py", line 250, in run_asgi
result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\fastapi\applications.py", line 276, in __call__
await super().__call__(scope, receive, send)
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\starlette\applications.py", line 122, in __call__
await self.middleware_stack(scope, receive, send)
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\starlette\middleware\errors.py", line 149, in __call__
await self.app(scope, receive, send)
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\starlette\middleware\cors.py", line 75, in __call__
await self.app(scope, receive, send)
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
raise exc
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
raise e
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
await self.app(scope, receive, send)
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\starlette\routing.py", line 718, in __call__
await route.handle(scope, receive, send)
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\starlette\routing.py", line 341, in handle
await self.app(scope, receive, send)
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\starlette\routing.py", line 82, in app
await func(session)
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\fastapi\routing.py", line 289, in app
await dependant.call(**values)
File "J:\k8s\ArgoCD\Git\Maui\The Path to Self-Transformation\Automation\screenshot-to-code\backend\routes\generate_code.py", line 234, in stream_code
prompt_messages, image_cache = await create_prompt(params, stack, input_mode)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\k8s\ArgoCD\Git\Maui\The Path to Self-Transformation\Automation\screenshot-to-code\backend\prompts\__init__.py", line 72, in create_prompt
prompt_messages = await assemble_claude_prompt_video(video_data_url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\k8s\ArgoCD\Git\Maui\The Path to Self-Transformation\Automation\screenshot-to-code\backend\video\utils.py", line 21, in assemble_claude_prompt_video
images = split_video_into_screenshots(video_data_url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\k8s\ArgoCD\Git\Maui\The Path to Self-Transformation\Automation\screenshot-to-code\backend\video\utils.py", line 79, in split_video_into_screenshots
clip = VideoFileClip(temp_video_file.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\moviepy\video\io\VideoFileClip.py", line 88, in __init__
self.reader = FFMPEG_VideoReader(filename, pix_fmt=pix_fmt,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\moviepy\video\io\ffmpeg_reader.py", line 35, in __init__
infos = ffmpeg_parse_infos(filename, print_infos, check_duration,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Greg\.virtualenvs\agents_with_json_mode_only-I1PZcyP6\Lib\site-packages\moviepy\video\io\ffmpeg_reader.py", line 289, in ffmpeg_parse_infos
raise IOError(("MoviePy error: failed to read the duration of file %s.\n"
OSError: MoviePy error: failed to read the duration of file C:\Users\Greg\AppData\Local\Temp\tmpkwxwbc8s.mp4.
Here are the file infos returned by ffmpeg:

ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 9.2.1 (GCC) 20200122
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
C:\Users\Greg\AppData\Local\Temp\tmpkwxwbc8s.mp4: Permission denied

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.