aipotheosis-labs / aipotheosis-labs/aci

Add three-tier difficulty prompts for user intent evaluation

Offen
#473 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @hanyixxx Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
4.9k
Forks
484
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

### Required Pre-requisites

- [x] I have read the [Documentation](https://www.aci.dev/docs)
- [x] I have searched the [Issue Tracker](https://github.com/aipotheosis-labs/aci/issues) and [Discussions](https://github.com/aipotheosis-labs/aci/discussions) that this hasn't been reported yet.
- [x] Consider asking in [Discussions](https://github.com/aipotheosis-labs/aci/discussions) first

### Motivation

Motivation: Create a robust evaluation framework with three distinct difficulty levels of synthetic data generation to comprehensively test the user intent search pipeline's performance across various real-world scenarios.

### Proposed Solution

## 🛠️ Proposed Solution

**Implement a three-tier synthetic data generation system with progressive difficulty levels:**

### Core Implementation:
1. **Three Prompt Templates** (`prompt_easy`, `prompt_medium`, `prompt_hard`)
- **Easy**: Direct app mentions + clear goals
- **Medium**: App mentions + business context, no explicit function terms
- **Hard**: Implicit/contextual language, optional app mentions

2. **Graduated Complexity Testing**
- Baseline performance (easy) → Real-world scenarios (medium) → Edge cases (hard)
- Systematic evaluation across user communication styles

3. **Enhanced Evaluation Pipeline**
- Custom dataset naming by difficulty level
- Comparative performance analysis
- Comprehensive metrics tracking per tier

### Outcome:
A systematic evaluation framework that provides **data-driven insights** into search pipeline performance across realistic user complexity scenarios, enabling targeted improvements and benchmarking.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.