elastic / elastic/relevance-studio

[Feature]: Batch API endpoint for creating judgements

Open
#35 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
42
Forks
18
Avg merge
1d 7h
Merged PRs (30d)
4

Description

### Problem Statement

When creating multiple judgements, especially using an LLM and MCP, sending individual create judgement requests becomes inefficient.

### Proposed Solution

This feature would add a batch create to the [judgements API](https://elastic.github.io/relevance-studio/#/docs/latest/reference/rest-api?id=judgements-api).

One solution would be to allow an alternate payload format in the existing endpoint, e.g. :

```
{
"judgements": [
{
"scenario_id": "67d554e2-8622-4c6c-9601-bf0e0a06128d",
"index": "demo",
"doc_id": "123",
"rating": 3
}
]
}
```

### Alternatives Considered

_No response_

### Enhancement Area

Frontend (UI)

### Priority

Critical - Blocking my workflow

### Use Case

_No response_

### Additional Context

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the judgements API documentation and the existing create-judgement endpoint. Trace its request validation and tests, then determine how the proposed judgements array should be accepted and verified. Done means one request creates all supplied judgements with the documented payload and coverage for the batch behavior.

Written by the indexing model from the issue text.

Assessment

Domain
api, frontend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.