abetlen / abetlen/llama-cpp-python
Images can not be decoded when using image_url with webp base64 from LibreChat
- Langage dominant
- Python
- Étoiles
- 10.6k
- Forks
- 1.4k
- Métriques de merge des PR
- Métriques de PR en attente
Description
# Expected Behavior
I was trying to integrate llama-cpp-python with LibreChat, has both use OpenAI API to communicate. The API implementation authorizes the sending of images with `image_url` that can be embedded in the request body. I was expecting the image to be analyzed by llava but I have an decoding error instead.
# Current Behavior
The request is correctly received by llama-cpp-python but I have the following error in logs:
```
clip_image_load_from_bytes: failed to decode image bytes
llava_image_embed_make_with_bytes: can't load image from bytes, is it a valid image?Segmentation fault (core dumped)
```
# Environment and Context
* Hardware
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7V13 64-Core Processor
CPU family: 25
Model: 1
Thread(s) per core: 1
Core(s) per socket: 24
Socket(s): 1
Stepping: 1
BogoMIPS: 4890.88
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable
nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalig
nsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero
xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm
Virtualization features:
Hypervisor vendor: Microsoft
Virtualization type: full
Caches (sum of all):
L1d: 768 KiB (24 instances)
L1i: 768 KiB (24 instances)
L2: 12 MiB (24 instances)
L3: 96 MiB (3 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-23
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec rstack overflow: Mitigation; safe RET, no microcode
Spec store bypass: Vulnerable
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected
```
* Operating System
`Linux *** 6.2.0-1019-azure #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux`
* SDK version (docker - cuda version compiled on the 2024/01/28:
```
Python 3.10.12
GNU Make 4.3
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
```
# Steps to Reproduce
I run a docker instance of llama-cpp-python behind a reverse proxy (for SSL handling) and I tried to send a completion request from LibreChat. Here is the request intercepted by my nginx reverse proxy:
```
"POST /v1/chat/completions HTTP/1.1" 502 157 "-" "OpenAI/JS 4.20.1" "-" "{\x0A \x22model\x22: \x22gpt-4-vision-preview\x22,\x0A \x22temperature\x22: 1,\x0A \x22top_p\x22: 1,\x0A \x22stream\x22: true,\x0A \x22messages\x22: [\x0A {\x0A \x22role\x22: \x22user\x22,\x0A \x22content\x22: [\x0A {\x0A \x22type\x22: \x22text\x22,\x0A \x22text\x22: \x22Que vois-tu ?\x22\x0A },\x0A {\x0A \x22type\x22: \x22image_url\x22,\x0A \x22image_url\x22: {\x0A \x22url\x22: \x22data:image/webp;base64,UklGRpKLAABXRUJQVlA4IIaLAABQTgKdASq8AjwDPm00lkikIqKqItIaeUANiWdu3NFmr2+8qtTVSszHwIS+fn3O2WGxmO6TE8Lw+biWIPMrzbffzudZint883zSeN8gTm9eAZ6/vXIR8Xb1E8bN7pn/O9gX7CewZ/dP9V1wnoiftV6b3tQfuF+2GYB/P/Kf9B/oP9Z4c/l/1P+W/vP+e/4X+K+iH9p/0fJV9N4F/Yj9z/k/3a/xX0J/r/+r/ovHn9d/a/+X/iP83+2vyHfln9C/xv9w/cP+//vJ70O5G3j/Uf9D/H+wd7K/Tv9v/bf85+2vyP/Xf9T/Resf81/n/+V7g/63/73+5eef40f4z/h+wp/RP79/4f8t/qfiM/wf/n/u/za+LH1d/6/9R8H39B/wn/e9db2t/ur7TxB+YMYxnrb3BqtCoqqpmWofkX055xxZM9zhyY+msgAaC5Q+yQwFkXTmRnJ8rnBi1xCJggO0a+vpFfoeGiRVcdTDavfafMG1e+09e5bEUvK2+S08//z2C95lQ8e4+lHfTo1wk1UFdvS+7oODKqqPMG1e+0+YNq99p8wbQtG5cHP7/3HPKV/CjpUeZ2H8K07BlsagelFTMlXDjqYbV77T5g2r32mA/t/cUH2jPd7L1lgHf79TR5dAF0a4FvPDADcWryKmVCgXCkp7q2xzNC1W4omhEK3dHqGVvyZ/GfMG1e+0+YNq99p6wV28ySOerKwj1ekwoYrm4AHBudLZ0S3+LDIsCVhEU+HMFq0uAr+fvfD509flRidWEhM6HAf0oLASmhcd2MFmoFUcGxSBCX7IjHX+4LF3SAPoTvRJbZPykdZvaeQvWQr32nzBtXvtPmDatbd/LgoKDWcfcJEtVtU5bAXy4rkpDK2zUp65UMw0rHgs3NImLw1sg5IO9tk6rJbYXUcBp7q9Xx0WZDT5g2r32nzBtXvsWJaEYEWdNg++z18E86yzBibwq/MjJKPOkGsHufhX3h++XAP7NJ0+7RhuxE0n9nPXrrBYd03Gw7GAoCEKO3nYSyWbtSaIjBlTIGWyON6KlusL5Kr7ac99XfI4KMMQ6rMAUwkALJRhDV2Yq/bDaHA3e+nS7Q/cX64JKtGqOrKaoh/YmEWB+0sEgNuGHQby5LzEoMWMLoL3h0uOULfsIXIIFiIoVV0AjLwhAqbcw6LZIA0sXiRxxwVGJG9bm6eTsr5fvzWy62yzqaoiEiRMRN/+Rl8zu...
```
As you can see the `image_url` is send in base 64 but somehow there is an issue when it is being decoded.
I have tried to see in the code where this action is happening but I would be glad if someone could help me find where this is comming from.
Thank you for your support!
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.