elastic / elastic/elastic-evals-sdk-python

KibanaInferenceClient sends no Authorization header (401 on real Kibana)

Offen
#8 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Python
Sterne
2
Forks
0
Ø Merge
1 T. 13 Std.
Gemergte PRs (30 T.)
18

Beschreibung

## Summary

`KibanaInferenceClient` (`elastic_evals/inference/client.py`) never sends an `Authorization` header on its requests, unlike `KibanaDatasetsClient` / `KibanaScoresClient` (`elastic_evals/api/datasets_client.py` / `scores_client.py`), which correctly build one via the shared `build_kibana_headers(api_key)` helper in `elastic_evals/api/headers.py`.

This means any evaluator built on `KibanaInferenceClient` — including `create_criteria_evaluator`, i.e. the LLM-as-judge evaluator path — fails with `401 Unauthorized` against a real, authenticated Kibana.

## Where

- `KibanaInferenceClient.chat_completion()` — headers built at client.py:165
- `KibanaInferenceClient.prompt()` — headers built at client.py:228

Both build:

```python
headers = {
"kbn-xsrf": "true",
"Content-Type": "application/json",
"x-elastic-internal-origin": "true",
**propagated_headers(),
}
```

with no `Authorization` entry, even though the client is constructed with a `connector_id` and clearly targets an authenticated Kibana instance.

Contrast with `elastic_evals/api/headers.py`:

```python
def build_kibana_headers(api_key: str | None) -> dict[str, str]:
headers = {
"Content-Type": "application/json",
"kbn-xsrf": "true",
"x-elastic-internal-origin": "true",
"Elastic-Api-Version": EVALS_API_VERSION,
}
if api_key:
headers["Authorization"] = f"ApiKey {api_key}"
return headers
```

which `KibanaDatasetsClient` and `KibanaScoresClient` both use correctly. `KibanaInferenceClient` doesn't call this helper (and isn't even constructed with an `api_key` today — only `kibana_url` and `connector_id`).

## Repro

1. Construct a `KibanaInferenceClient(kibana_url=..., connector_id=...)` pointed at a real Kibana instance.
2. Call `.prompt(...)` (directly, or indirectly via `create_criteria_evaluator(...).evaluate(...)`).
3. Request fails with `401 Unauthorized`:
```
elastic_evals.inference.client.KibanaInferenceError: Kibana inference prompt request failed with 401: {"statusCode": 401, "error": "Unauthorized", "message": "Unauthorized"}
```

## Environment

- `elastic-evals` version `0.1.0` (pip)

## Impact

Blocks any use of LLM-as-judge evaluators (`create_criteria_evaluator` and anything else built on `KibanaInferenceClient`) via the Python SDK against a real Kibana — not specific to one caller's setup.

## Suggested fix

Give `KibanaInferenceClient` an `api_key` (mirroring `KibanaDatasetsClient`/`KibanaScoresClient`) and route its header construction through `build_kibana_headers(api_key)` (or an equivalent), same as the other two clients.

---
Found while live-testing an LLM-as-judge evaluator against a real Kibana connector; worked around it in our own test harness only (a local subclass overriding `prompt()` to add the header) — not a fix, just enough to validate the evaluator itself.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Beginne mit dem Konstruktor von KibanaInferenceClient sowie chat_completion() und prompt() in elastic_evals/inference/client.py und vergleiche anschließend deren Header mit build_kibana_headers() in elastic_evals/api/headers.py sowie mit den Datasets- und Scores-Clients. Als erledigt gilt die Aufgabe, wenn der Client den API key akzeptiert, authentifizierte Requests den erwarteten Authorization-Header enthalten und der echte Kibana-Inference-Pfad nicht mehr mit 401 Unauthorized fehlschlägt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
api, backend
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Ruhig
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
76/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.