elastic / elastic/elastic-evals-sdk-python

KibanaInferenceClient sends no Authorization header (401 on real Kibana)

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
18

Description

## Summary

`KibanaInferenceClient` (`elastic_evals/inference/client.py`) never sends an `Authorization` header on its requests, unlike `KibanaDatasetsClient` / `KibanaScoresClient` (`elastic_evals/api/datasets_client.py` / `scores_client.py`), which correctly build one via the shared `build_kibana_headers(api_key)` helper in `elastic_evals/api/headers.py`.

This means any evaluator built on `KibanaInferenceClient` — including `create_criteria_evaluator`, i.e. the LLM-as-judge evaluator path — fails with `401 Unauthorized` against a real, authenticated Kibana.

## Where

- `KibanaInferenceClient.chat_completion()` — headers built at client.py:165
- `KibanaInferenceClient.prompt()` — headers built at client.py:228

Both build:

```python
headers = {
"kbn-xsrf": "true",
"Content-Type": "application/json",
"x-elastic-internal-origin": "true",
**propagated_headers(),
}
```

with no `Authorization` entry, even though the client is constructed with a `connector_id` and clearly targets an authenticated Kibana instance.

Contrast with `elastic_evals/api/headers.py`:

```python
def build_kibana_headers(api_key: str | None) -> dict[str, str]:
headers = {
"Content-Type": "application/json",
"kbn-xsrf": "true",
"x-elastic-internal-origin": "true",
"Elastic-Api-Version": EVALS_API_VERSION,
}
if api_key:
headers["Authorization"] = f"ApiKey {api_key}"
return headers
```

which `KibanaDatasetsClient` and `KibanaScoresClient` both use correctly. `KibanaInferenceClient` doesn't call this helper (and isn't even constructed with an `api_key` today — only `kibana_url` and `connector_id`).

## Repro

1. Construct a `KibanaInferenceClient(kibana_url=..., connector_id=...)` pointed at a real Kibana instance.
2. Call `.prompt(...)` (directly, or indirectly via `create_criteria_evaluator(...).evaluate(...)`).
3. Request fails with `401 Unauthorized`:
```
elastic_evals.inference.client.KibanaInferenceError: Kibana inference prompt request failed with 401: {"statusCode": 401, "error": "Unauthorized", "message": "Unauthorized"}
```

## Environment

- `elastic-evals` version `0.1.0` (pip)

## Impact

Blocks any use of LLM-as-judge evaluators (`create_criteria_evaluator` and anything else built on `KibanaInferenceClient`) via the Python SDK against a real Kibana — not specific to one caller's setup.

## Suggested fix

Give `KibanaInferenceClient` an `api_key` (mirroring `KibanaDatasetsClient`/`KibanaScoresClient`) and route its header construction through `build_kibana_headers(api_key)` (or an equivalent), same as the other two clients.

---
Found while live-testing an LLM-as-judge evaluator against a real Kibana connector; worked around it in our own test harness only (a local subclass overriding `prompt()` to add the header) — not a fix, just enough to validate the evaluator itself.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.