Add result filtering to generate datasets matching specific criteria
- Dominant language
- Python
- Stars
- 41
- Forks
- 5
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
## Problem
When working with `Result` objects from benchmark runs, there is currently no built-in way to filter responses based on specific criteria. Users frequently need to generate subsets/datasets that match particular conditions, for example:
- Only successful responses (excluding errors)
- Only responses that used prompt caching
- Responses within certain latency bounds (TTFT, TTLT)
- Responses within certain token count ranges
The existing `get_dimension()` method supports a basic single-field equality filter, but it only extracts dimension values — it doesn't produce a new `Result` object. This means users cannot easily:
1. Remove error responses before computing statistics
2. Isolate cached vs. non-cached responses for comparison
3. Filter by numeric ranges (e.g. latency or token bounds)
4. Compose multiple filter criteria together
5. Compute fresh `stats` on a filtered subset
Users end up writing manual list comprehensions over `result.responses` and reconstructing `Result` objects by hand, which is error-prone and loses metadata.
## Use Cases
- Compare latency distributions for cached vs. non-cached requests
- Compute clean statistics after removing failed/error responses
- Generate datasets of only "fast" responses (e.g. TTFT < threshold)
- Filter to specific token count ranges for cost analysis
- Apply arbitrary domain-specific predicates for custom analysis
Contributor guide
Research direction
Start by locating the Result class, its existing get_dimension() method, and the responses collection. Define the filtering behavior around the listed equality, range, and predicate use cases while preserving metadata and producing fresh statistics. Done means callers can compose criteria and obtain a filtered Result without manually reconstructing it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100