elastic / elastic/elastic-evals-sdk-python

[kbn-evals] Tutorial-grade onboarding example

Open
#35 0 comments 0 reactions 1 assignee Claimed by @milistu View on GitHub
kbn-evals Team:nightshift-context-and-research
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
18

Description

### Summary

The current example runs a command and ends. It doesn't explain what was created in Kibana, what each score means, or what to look for. A user finishing the example doesn't know what they just did.

### What we want

An example that builds up from scratch: define a dataset, write a task, add evaluators, run it, then walk the user through what appeared in Kibana.

Reference bar: the [DSPy "Program, don't prompt" tutorial](https://dspy.ai/getting-started/program-dont-prompt/) (start small, add one concept at a time, run it, explain the result, move on). The [elasticsearch-labs quick-start notebook](https://colab.research.google.com/github/elastic/elasticsearch-labs/blob/main/notebooks/search/00-quick-start.ipynb) shows the credential-via-env-var pattern to follow.

### Format

Google Colab notebook with an "Open in Colab" badge in the repo. Users run it without installing anything locally. All connection details come from environment variables set in the first cell.

### Connectivity options

The notebook should work with both setups, selected by which env vars the user sets:

**Option A: Elastic Cloud (recommended for Colab)**
A free trial at cloud.elastic.co gives a full Elasticsearch + Kibana stack. The user sets `ELASTIC_CLOUD_URL` and `ELASTIC_API_KEY`. No local installation required.

**Option B: Local stack**
The user runs Elasticsearch and Kibana locally and passes `KIBANA_URL` and `ELASTIC_API_KEY`. The notebook connects to their local stack. No Colab-side setup is needed.

The first cell of the notebook should detect which env vars are present and print a clear message if neither set is found.

### Done when

- [ ] A Colab notebook guides the user from an empty file to a completed run with results in Kibana
- [ ] Each section explains what it does and what to look for after running it
- [ ] The notebook works with Elastic Cloud credentials and with local stack credentials
- [ ] An "Open in Colab" badge appears in the examples README

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.