elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Documentation: prerequisites, quick start, env vars, evaluator reference
- Lingua principale
- Python
- Stelle
- 2
- Fork
- 0
- Merge medio
- 1g 13h
- PR unite (30g)
- 18
Descrizione
### Summary
Several gaps in the getting-started experience that block new users from running a first experiment.
### Principle
The README should link to existing Elastic docs rather than duplicate them. If the connector creation flow changes, that change should happen in Kibana docs, not here. One sentence per concept, one link for depth.
### Problems
1. No prerequisites section: users don't know they need a Kibana connector before running anything (`README.md`)
2. Missing connector raises `KeyError` with doubled quotes and no fix hint (`config.py:21-25`)
3. Env var table has three problems: `ELASTIC_EVALS_TRACING_EXPORTER` is documented but never read by `from_env()` so setting it as an env var has no effect (only the CLI flag works); `ELASTIC_EVALS_SUITE_ID` is read but undocumented; `EDOT_ENDPOINT` is missing from the table. The default Kibana port is `5601` in `config.py` but Quick start uses `5620` and examples vary between the two (`config.py:72-107`)
4. The evaluator reference omits `kibana_evaluators`, `KibanaEvaluatorConfig`, and all four CODE evaluators. About 29% of the README is a low-level API smoke test with no explanatory value. The Correctness evaluator docs describe 3 sub-scores but 4 rows appear in Kibana (the qualitative analysis row is always emitted alongside them)
5. Quick start doesn't mention that each run replaces the dataset on Kibana
6. `examples/README.md` is a one-line placeholder. `claude_code_eval` is not listed in the main README examples section
7. `examples/claude_code_eval/README.md` mixes host-side `uv run` with Docker-only hostnames. A real connector ID is hardcoded at line 32. The URL includes an unexplained `/dev` path segment that causes 404s on a standard Kibana setup
### Done when
- [ ] Prerequisites section explains what a connector is, links to Kibana docs for creation steps, and states the minimum stack version
- [ ] Missing connector raises a `ConfigurationError` with a clear message and a link to the prerequisites section
- [ ] Env var table matches what `from_env()` actually reads (no undocumented vars, no dead vars); `ELASTIC_EVALS_TRACING_EXPORTER` is either wired into `from_env()` or removed from the table
- [ ] Default Kibana port is consistent across all docs and examples
- [ ] Evaluator reference covers all built-in evaluators and the composable API; Correctness evaluator section explains the 4-row structure in Kibana
- [ ] Quick start notes that running an experiment replaces the existing dataset
- [ ] `examples/README.md` is filled in and `claude_code_eval` appears in the main README
- [ ] `claude_code_eval` README uses `localhost`, a `CONNECTOR_ID` placeholder, and explains the `/dev` basePath or removes it
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Inizia da README.md e config.py:21-25 e 72-107, quindi esamina examples/README.md e examples/claude_code_eval/README.md. Confronta la documentazione con from_env(), le API dell'evaluator e gli esempi elencati. Il lavoro è completato quando ogni elemento della checklist è trattato in modo coerente in README, nel comportamento della configurazione e nelle istruzioni degli esempi.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- documentation
- Tipo di issue
- Documentazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Specificata chiaramente
- Idoneità per principianti
- 48/100