aipotheosis-labs / aipotheosis-labs/aci

Add three-tier difficulty prompts for user intent evaluation

Ouverte
#473 0 commentaires 0 réactions 1 personne assignée Réclamée par @hanyixxx Voir sur GitHub
Langage dominant
Python
Étoiles
4.9k
Forks
484
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

### Required Pre-requisites

- [x] I have read the [Documentation](https://www.aci.dev/docs)
- [x] I have searched the [Issue Tracker](https://github.com/aipotheosis-labs/aci/issues) and [Discussions](https://github.com/aipotheosis-labs/aci/discussions) that this hasn't been reported yet.
- [x] Consider asking in [Discussions](https://github.com/aipotheosis-labs/aci/discussions) first

### Motivation

Motivation: Create a robust evaluation framework with three distinct difficulty levels of synthetic data generation to comprehensively test the user intent search pipeline's performance across various real-world scenarios.

### Proposed Solution

## 🛠️ Proposed Solution

**Implement a three-tier synthetic data generation system with progressive difficulty levels:**

### Core Implementation:
1. **Three Prompt Templates** (`prompt_easy`, `prompt_medium`, `prompt_hard`)
- **Easy**: Direct app mentions + clear goals
- **Medium**: App mentions + business context, no explicit function terms
- **Hard**: Implicit/contextual language, optional app mentions

2. **Graduated Complexity Testing**
- Baseline performance (easy) → Real-world scenarios (medium) → Edge cases (hard)
- Systematic evaluation across user communication styles

3. **Enhanced Evaluation Pipeline**
- Custom dataset naming by difficulty level
- Comparative performance analysis
- Comprehensive metrics tracking per tier

### Outcome:
A systematic evaluation framework that provides **data-driven insights** into search pipeline performance across realistic user complexity scenarios, enabling targeted improvements and benchmarking.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.