awslabs / awslabs/llmeter

Add result filtering to generate datasets matching specific criteria

Open
#86 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
41
Forks
5
Avg merge
2m
Merged PRs (30d)
1

Description

## Problem

When working with `Result` objects from benchmark runs, there is currently no built-in way to filter responses based on specific criteria. Users frequently need to generate subsets/datasets that match particular conditions, for example:

- Only successful responses (excluding errors)
- Only responses that used prompt caching
- Responses within certain latency bounds (TTFT, TTLT)
- Responses within certain token count ranges

The existing `get_dimension()` method supports a basic single-field equality filter, but it only extracts dimension values — it doesn't produce a new `Result` object. This means users cannot easily:

1. Remove error responses before computing statistics
2. Isolate cached vs. non-cached responses for comparison
3. Filter by numeric ranges (e.g. latency or token bounds)
4. Compose multiple filter criteria together
5. Compute fresh `stats` on a filtered subset

Users end up writing manual list comprehensions over `result.responses` and reconstructing `Result` objects by hand, which is error-prone and loses metadata.

## Use Cases

- Compare latency distributions for cached vs. non-cached requests
- Compute clean statistics after removing failed/error responses
- Generate datasets of only "fast" responses (e.g. TTFT < threshold)
- Filter to specific token count ranges for cost analysis
- Apply arbitrary domain-specific predicates for custom analysis

Contributor guide

Open the contributing guide

Research direction

Start by locating the Result class, its existing get_dimension() method, and the responses collection. Define the filtering behavior around the listed equality, range, and predicate use cases while preserving metadata and producing fresh statistics. Done means callers can compose criteria and obtain a filtered Result without manually reconstructing it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.