google / google/adk-python

Proxied file uploads bypass the LiteLLM Proxy and are sent to api.openai.com

Đang mở
#6,722 4 bình luận 0 reaction 1 người được giao Được @surajksharma07 nhận Xem trên GitHub
models request clarification
Ngôn ngữ chính
Python
Star
21.5k
Fork
4k
Merge trung bình
1 ngày 14 giờ
Pull request đã merge (30 ngày)
37

Mô tả

**Describe the bug**

When a model is served through a LiteLLM Proxy, ADK uploads file content (PDFs
and the other types in `_SUPPORTED_FILE_CONTENT_MIME_TYPES`) to an endpoint
resolved independently of the completion call. The completion goes to the
proxy. The upload does not.

For `azure` this fails locally with a missing credentials error. For `openai`
it does not fail at all. `get_openai_credentials` defaults `api_base` to
`https://api.openai.com/v1`, so if an `OPENAI_API_KEY` is present in the
environment the upload succeeds against the public OpenAI API. File content
meant for a self hosted or vendored proxy leaves it, and the developer's own
OpenAI key is what sends it.

That is the case I care about here. A proxy is usually deployed precisely
because the data is not supposed to reach the provider directly.

**To Reproduce**

`.env`, which is a plausible developer setup: proxy credentials plus a personal
OpenAI key left over from other work.

```
LITELLM_PROXY_API_BASE=http://127.0.0.1:8877
LITELLM_PROXY_API_KEY=sk-proxy-secret
OPENAI_API_KEY=sk-personal-developer-key
```

`repro.py`. The local server stands in for the proxy so we can see what it
receives, and the httpx patch records anything that leaves localhost instead of
sending it.

```python
import asyncio, json, threading
from http.server import BaseHTTPRequestHandler, HTTPServer

from dotenv import load_dotenv
load_dotenv()

MODEL = "litellm_proxy/openai/gpt-4o"

PROXY_RECEIVED = []

class _Proxy(BaseHTTPRequestHandler):

def do_POST(self):
n = int(self.headers.get("Content-Length") or 0)
self.rfile.read(n)
PROXY_RECEIVED.append(self.path)
if "files" in self.path:
payload = {"id": "file-via-proxy", "object": "file", "bytes": n,
"created_at": 0, "filename": "f.pdf",
"purpose": "assistants", "status": "processed"}
else:
payload = {"id": "chatcmpl-1", "object": "chat.completion", "created": 0,
"model": "m", "choices": [{"index": 0, "finish_reason": "stop",
"message": {"role": "assistant", "content": "ok"}}],
"usage": {"prompt_tokens": 1, "completion_tokens": 1,
"total_tokens": 2}}
body = json.dumps(payload).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)

def log_message(self, *a):
pass

HTTPServer.allow_reuse_address = True
threading.Thread(
target=HTTPServer(("127.0.0.1", 8877), _Proxy).serve_forever,
daemon=True).start()

import httpx
LEFT_THE_PROXY = []
_orig_send = httpx.AsyncClient.send

async def _spy(self, request, *args, **kwargs):
if request.url.host not in ("127.0.0.1", "localhost"):
LEFT_THE_PROXY.append(str(request.url))
raise RuntimeError(f"blocked egress to {request.url}")
return await _orig_send(self, request, *args, **kwargs)

httpx.AsyncClient.send = _spy

from google.adk.models.lite_llm import LiteLlm
from google.adk.models.llm_request import LlmRequest
from google.genai import types

llm = LiteLlm(model=MODEL)
request = LlmRequest(model=MODEL, contents=[types.Content(role="user", parts=[
types.Part(text="summarize this"),
types.Part.from_bytes(data=b"%PDF-1.4 confidential",
mime_type="application/pdf"),
])])

async def main():
try:
async for _ in llm.generate_content_async(request, stream=False):
pass
except Exception as e:
print(f"raised : {type(e).__name__}: {str(e)[:100]}")
print(f"proxy received : {PROXY_RECEIVED or []}")
print(f"left the proxy : {LEFT_THE_PROXY or []}")

asyncio.run(main())
```

Output on `main` at 370027a7:

```
raised : APIConnectionError: Connection error.
proxy received : []
left the proxy : ['https://api.openai.com/v1/files', ...]
```

The proxy receives nothing. The upload goes to OpenAI. It only surfaces as a
connection error because the script blocks the request. Without the block it
succeeds, using `OPENAI_API_KEY`.

**Expected behavior**

The upload goes wherever the completion goes. If the proxy endpoint cannot be
determined, the upload fails instead of falling back to a provider default.

**Three configurations reach this**

All three were run with the script above, changing only `.env` and `MODEL`.

| # | Configuration | Model | `proxy received` | `left the proxy` |
|---|---|---|---|---|
| 1 | `LITELLM_PROXY_API_BASE` + `LITELLM_PROXY_API_KEY` | `litellm_proxy/openai/gpt-4o` | `[]` | `api.openai.com/v1/files` |
| 2 | `USE_LITELLM_PROXY=true` | `openai/gpt-4o` | `[]` | `api.openai.com/v1/files` |
| 3 | neither, only `OPENAI_API_KEY` | `litellm_proxy/openai/gpt-4o` | `[]` | `api.openai.com/v1/files` |

Case 2 is worth spelling out. LiteLLM supports routing unprefixed models
through the proxy with `USE_LITELLM_PROXY`, and its own docstring says that
flag exists for Google ADK (BerriAI/litellm#10559). A model string of
`openai/gpt-4o` gives no hint that it is proxied, so anything matching on the
`litellm_proxy/` prefix alone will miss it.

For reference, LiteLLM resolves the proxy endpoint for the completion call in
`litellm/llms/litellm_proxy/chat/transformation.py`:

```python
api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
dynamic_api_key = api_key or get_secret_str("LITELLM_PROXY_API_KEY")
```

`litellm.acreate_file` does not consult those, so unless ADK forwards an
endpoint the two halves of the request diverge.

**Relation to #6538 and #6578**

#6578 (merged as 602e58d) fixed the payload shape for proxied models. Nested
`litellm_proxy/azure/...` identifiers now resolve to the underlying provider,
so the upload path runs instead of emitting an inline `file_data` block. That
part is correct and nothing here contests it.

The same commit also set `custom_llm_provider="openai"` for proxied uploads.
That is the line that turns cases 1 and 3 from a local failure into a request
to `api.openai.com`, because `openai` is the one provider whose credential
resolution defaults to the public API:

```python
# litellm/llms/openai/common_utils.py
resolved_api_base = (
api_base or litellm.api_base or os.getenv("OPENAI_BASE_URL")
or os.getenv("OPENAI_API_BASE") or "https://api.openai.com/v1"
)
```

On `main`, `_get_content` takes only `parts`, `provider` and `model`. There is
no parameter through which a proxy endpoint could arrive, and ADK never sets
`litellm.api_base`. So there is no configuration in which that override reaches
a proxy. It resolves to the public endpoint every time.

**Environment**

- ADK `main` at 370027a7. The behavior starts at 602e58d.
- litellm 1.85.7, and also reproduced on 1.84.0, the floor of the
`litellm>=1.84` constraint in `pyproject.toml`. In both,
`OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS` is `{openai, hosted_vllm}` and
`create_file` has no `litellm_proxy` branch, so routing the upload as
`litellm_proxy` is not an option either.
- Python 3.11, macOS.

**Additional context**

I have a fix and will open a PR referencing this issue. One part of it is a
behavior change I would rather have decided explicitly than slipped into a
diff: when no proxy endpoint can be determined, the upload raises instead of
quietly using the provider default. For anyone who has `OPENAI_API_KEY` set
that converts a currently "working" upload into an error. I think an explicit
error is better than a silent misroute, but it is a trade off and it is your
call.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.