elastic / elastic/elastic-evals-sdk-python
[kbn-evals] connector_id reported as model identity
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 18
Description
### Summary
Kibana shows a connector ID as both the task model and the evaluator model. The actual models are available in telemetry. CODE evaluators (latency, tokens, tool_calls) don't use a model at all but still get one assigned.
### Problem
- `executor/client.py:256` falls back to `CONNECTOR_ID` as `task_model` in `_build_task_model()` when `ELASTIC_EVALS_MODEL` is not set
- `client.py:261-262` assigns the same connector ID as the evaluator model to all evaluators, including CODE evaluators
- A connector ID identifies a route to an inference service, not a model
### Fix
- Read the actual task model from execution data (e.g. `gen_ai.request.model` telemetry)
- Read the actual evaluator model from the Kibana evaluator response
- Don't assign a model field to CODE evaluators
- Make `CONNECTOR_ID` optional at the global config level; require it only when inference is actually used
### Done when
- [ ] Kibana shows the actual task model, not the connector ID
- [ ] CODE evaluators are ingested without a model field
- [ ] `CONNECTOR_ID` is not required for runs that don't use Kibana inference
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in executor/client.py around _build_task_model() at line 256 and the evaluator model assignment at lines 261-262. Trace how execution telemetry and Kibana evaluator responses provide model identity, and how CODE evaluators are represented. Done means actual task and evaluator models are reported, CODE evaluators have no model field, and runs not using Kibana inference do not require CONNECTOR_ID.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100