microsoft / microsoft/rag-experiment-accelerator
Separate each search type into its own mlflow run to allow comparison using Azure ML / mlflow
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 311
- Forks
- 111
- PR merge metrics
- No merged PRs in 30d
Description
AS a researcher I would like to be able to compare metric results across different types of search approaches in Azure ML / ML Flow
SO, I would be able to choose best type of search for my case.
related to #529
Note: currently eval step averages all metrics for all search types and log to mflow only the mean value per metric, but also uploads the full detailed table to azureml.
DoD:
- query and eval steps will run per search type in parallel (like Index does with
IndexConfigclass)
### Tasks
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the query and eval steps and the existing parallel execution based on the IndexConfig class first. Done means query and eval run separately for each search type in parallel, with metrics logged per type so results can be compared rather than only averaged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- data, machine-learning, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100