elastic / elastic/elastic-evals-sdk-python

[kbn-evals] HTTP efficiency: connection pooling and score batching

Open
#39 0 comments 0 reactions 0 assignees View on GitHub
kbn-evals Team:nightshift-context-and-research
Dominant language
Python
Stars
2
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
18

Description

### Summary

Every request to Kibana opens a new `httpx.AsyncClient`, makes one call, and closes it. For N examples x R repetitions, every score ingest is a separate TCP connection to the same host. The scores API supports batching but each evaluator result is sent alone.

The [httpx async docs](https://www.python-httpx.org/async/#opening-and-closing-clients) say: "do not instantiate multiple client instances inside a hot loop."

### Problem

- `datasets_client.py:61-62`, `scores_client.py:39`, `evaluators_client.py:83-88`, `inference/client.py:179-180` all use `async with httpx.AsyncClient(...) as client:` per request
- Score ingest sends one `POST` per evaluator result; the scores API accepts a list of scores per request

### Fix

- Create one `httpx.AsyncClient` per API client class in `__init__`, close it in `aclose()`
- `ElasticEvalsClient` manages the lifecycle via `async with` or a `finally` block
- Batch score ingest calls per example rather than per evaluator result

### Done when

- [ ] Each API client holds one `httpx.AsyncClient` for the lifetime of a run
- [ ] `ElasticEvalsClient` supports `async with client:` or `await client.aclose()`
- [ ] Score ingest sends one request per example instead of one per evaluator result

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.