elastic / elastic/elastic-evals-sdk-python
[kbn-evals] Documentation: prerequisites, quick start, env vars, evaluator reference
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 18
Description
### Summary
Several gaps in the getting-started experience that block new users from running a first experiment.
### Principle
The README should link to existing Elastic docs rather than duplicate them. If the connector creation flow changes, that change should happen in Kibana docs, not here. One sentence per concept, one link for depth.
### Problems
1. No prerequisites section: users don't know they need a Kibana connector before running anything (`README.md`)
2. Missing connector raises `KeyError` with doubled quotes and no fix hint (`config.py:21-25`)
3. Env var table has three problems: `ELASTIC_EVALS_TRACING_EXPORTER` is documented but never read by `from_env()` so setting it as an env var has no effect (only the CLI flag works); `ELASTIC_EVALS_SUITE_ID` is read but undocumented; `EDOT_ENDPOINT` is missing from the table. The default Kibana port is `5601` in `config.py` but Quick start uses `5620` and examples vary between the two (`config.py:72-107`)
4. The evaluator reference omits `kibana_evaluators`, `KibanaEvaluatorConfig`, and all four CODE evaluators. About 29% of the README is a low-level API smoke test with no explanatory value. The Correctness evaluator docs describe 3 sub-scores but 4 rows appear in Kibana (the qualitative analysis row is always emitted alongside them)
5. Quick start doesn't mention that each run replaces the dataset on Kibana
6. `examples/README.md` is a one-line placeholder. `claude_code_eval` is not listed in the main README examples section
7. `examples/claude_code_eval/README.md` mixes host-side `uv run` with Docker-only hostnames. A real connector ID is hardcoded at line 32. The URL includes an unexplained `/dev` path segment that causes 404s on a standard Kibana setup
### Done when
- [ ] Prerequisites section explains what a connector is, links to Kibana docs for creation steps, and states the minimum stack version
- [ ] Missing connector raises a `ConfigurationError` with a clear message and a link to the prerequisites section
- [ ] Env var table matches what `from_env()` actually reads (no undocumented vars, no dead vars); `ELASTIC_EVALS_TRACING_EXPORTER` is either wired into `from_env()` or removed from the table
- [ ] Default Kibana port is consistent across all docs and examples
- [ ] Evaluator reference covers all built-in evaluators and the composable API; Correctness evaluator section explains the 4-row structure in Kibana
- [ ] Quick start notes that running an experiment replaces the existing dataset
- [ ] `examples/README.md` is filled in and `claude_code_eval` appears in the main README
- [ ] `claude_code_eval` README uses `localhost`, a `CONNECTOR_ID` placeholder, and explains the `/dev` basePath or removes it
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.