google / google/adk-python

Proxied file uploads bypass the LiteLLM Proxy and are sent to api.openai.com

未关闭
#6,722 4 条评论 0 个 reaction 已指派 1 人 已被 @surajksharma07 认领 在 GitHub 查看
models request clarification
主要语言
Python
星标
21.5k
派生
4k
平均合并
1 天 14 小时
30 天内合并 PR
37

描述

**Describe the bug**

When a model is served through a LiteLLM Proxy, ADK uploads file content (PDFs
and the other types in `_SUPPORTED_FILE_CONTENT_MIME_TYPES`) to an endpoint
resolved independently of the completion call. The completion goes to the
proxy. The upload does not.

For `azure` this fails locally with a missing credentials error. For `openai`
it does not fail at all. `get_openai_credentials` defaults `api_base` to
`https://api.openai.com/v1`, so if an `OPENAI_API_KEY` is present in the
environment the upload succeeds against the public OpenAI API. File content
meant for a self hosted or vendored proxy leaves it, and the developer's own
OpenAI key is what sends it.

That is the case I care about here. A proxy is usually deployed precisely
because the data is not supposed to reach the provider directly.

**To Reproduce**

`.env`, which is a plausible developer setup: proxy credentials plus a personal
OpenAI key left over from other work.

```
LITELLM_PROXY_API_BASE=http://127.0.0.1:8877
LITELLM_PROXY_API_KEY=sk-proxy-secret
OPENAI_API_KEY=sk-personal-developer-key
```

`repro.py`. The local server stands in for the proxy so we can see what it
receives, and the httpx patch records anything that leaves localhost instead of
sending it.

```python
import asyncio, json, threading
from http.server import BaseHTTPRequestHandler, HTTPServer

from dotenv import load_dotenv
load_dotenv()

MODEL = "litellm_proxy/openai/gpt-4o"

PROXY_RECEIVED = []

class _Proxy(BaseHTTPRequestHandler):

def do_POST(self):
n = int(self.headers.get("Content-Length") or 0)
self.rfile.read(n)
PROXY_RECEIVED.append(self.path)
if "files" in self.path:
payload = {"id": "file-via-proxy", "object": "file", "bytes": n,
"created_at": 0, "filename": "f.pdf",
"purpose": "assistants", "status": "processed"}
else:
payload = {"id": "chatcmpl-1", "object": "chat.completion", "created": 0,
"model": "m", "choices": [{"index": 0, "finish_reason": "stop",
"message": {"role": "assistant", "content": "ok"}}],
"usage": {"prompt_tokens": 1, "completion_tokens": 1,
"total_tokens": 2}}
body = json.dumps(payload).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)

def log_message(self, *a):
pass

HTTPServer.allow_reuse_address = True
threading.Thread(
target=HTTPServer(("127.0.0.1", 8877), _Proxy).serve_forever,
daemon=True).start()

import httpx
LEFT_THE_PROXY = []
_orig_send = httpx.AsyncClient.send

async def _spy(self, request, *args, **kwargs):
if request.url.host not in ("127.0.0.1", "localhost"):
LEFT_THE_PROXY.append(str(request.url))
raise RuntimeError(f"blocked egress to {request.url}")
return await _orig_send(self, request, *args, **kwargs)

httpx.AsyncClient.send = _spy

from google.adk.models.lite_llm import LiteLlm
from google.adk.models.llm_request import LlmRequest
from google.genai import types

llm = LiteLlm(model=MODEL)
request = LlmRequest(model=MODEL, contents=[types.Content(role="user", parts=[
types.Part(text="summarize this"),
types.Part.from_bytes(data=b"%PDF-1.4 confidential",
mime_type="application/pdf"),
])])

async def main():
try:
async for _ in llm.generate_content_async(request, stream=False):
pass
except Exception as e:
print(f"raised : {type(e).__name__}: {str(e)[:100]}")
print(f"proxy received : {PROXY_RECEIVED or []}")
print(f"left the proxy : {LEFT_THE_PROXY or []}")

asyncio.run(main())
```

Output on `main` at 370027a7:

```
raised : APIConnectionError: Connection error.
proxy received : []
left the proxy : ['https://api.openai.com/v1/files', ...]
```

The proxy receives nothing. The upload goes to OpenAI. It only surfaces as a
connection error because the script blocks the request. Without the block it
succeeds, using `OPENAI_API_KEY`.

**Expected behavior**

The upload goes wherever the completion goes. If the proxy endpoint cannot be
determined, the upload fails instead of falling back to a provider default.

**Three configurations reach this**

All three were run with the script above, changing only `.env` and `MODEL`.

| # | Configuration | Model | `proxy received` | `left the proxy` |
|---|---|---|---|---|
| 1 | `LITELLM_PROXY_API_BASE` + `LITELLM_PROXY_API_KEY` | `litellm_proxy/openai/gpt-4o` | `[]` | `api.openai.com/v1/files` |
| 2 | `USE_LITELLM_PROXY=true` | `openai/gpt-4o` | `[]` | `api.openai.com/v1/files` |
| 3 | neither, only `OPENAI_API_KEY` | `litellm_proxy/openai/gpt-4o` | `[]` | `api.openai.com/v1/files` |

Case 2 is worth spelling out. LiteLLM supports routing unprefixed models
through the proxy with `USE_LITELLM_PROXY`, and its own docstring says that
flag exists for Google ADK (BerriAI/litellm#10559). A model string of
`openai/gpt-4o` gives no hint that it is proxied, so anything matching on the
`litellm_proxy/` prefix alone will miss it.

For reference, LiteLLM resolves the proxy endpoint for the completion call in
`litellm/llms/litellm_proxy/chat/transformation.py`:

```python
api_base = api_base or get_secret_str("LITELLM_PROXY_API_BASE")
dynamic_api_key = api_key or get_secret_str("LITELLM_PROXY_API_KEY")
```

`litellm.acreate_file` does not consult those, so unless ADK forwards an
endpoint the two halves of the request diverge.

**Relation to #6538 and #6578**

#6578 (merged as 602e58d) fixed the payload shape for proxied models. Nested
`litellm_proxy/azure/...` identifiers now resolve to the underlying provider,
so the upload path runs instead of emitting an inline `file_data` block. That
part is correct and nothing here contests it.

The same commit also set `custom_llm_provider="openai"` for proxied uploads.
That is the line that turns cases 1 and 3 from a local failure into a request
to `api.openai.com`, because `openai` is the one provider whose credential
resolution defaults to the public API:

```python
# litellm/llms/openai/common_utils.py
resolved_api_base = (
api_base or litellm.api_base or os.getenv("OPENAI_BASE_URL")
or os.getenv("OPENAI_API_BASE") or "https://api.openai.com/v1"
)
```

On `main`, `_get_content` takes only `parts`, `provider` and `model`. There is
no parameter through which a proxy endpoint could arrive, and ADK never sets
`litellm.api_base`. So there is no configuration in which that override reaches
a proxy. It resolves to the public endpoint every time.

**Environment**

- ADK `main` at 370027a7. The behavior starts at 602e58d.
- litellm 1.85.7, and also reproduced on 1.84.0, the floor of the
`litellm>=1.84` constraint in `pyproject.toml`. In both,
`OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS` is `{openai, hosted_vllm}` and
`create_file` has no `litellm_proxy` branch, so routing the upload as
`litellm_proxy` is not an option either.
- Python 3.11, macOS.

**Additional context**

I have a fix and will open a PR referencing this issue. One part of it is a
behavior change I would rather have decided explicitly than slipped into a
diff: when no proxy endpoint can be determined, the upload raises instead of
quietly using the provider default. For anyone who has `OPENAI_API_KEY` set
that converts a currently "working" upload into an error. I think an explicit
error is better than a silent misroute, but it is a trade off and it is your
call.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。