InseeFrLab / InseeFrLab/auto-tuning-vllm

Implement a Fixed Schedule dataset from production traces distribution

Open
#9 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
0
Avg merge
34m
Merged PRs (30d)
2

Description

As a user, i would like to generate a dataset in this format :

`{"timestamp": 0, "input_length": 100, "output_length": 200, "hash_ids": [1001]}
{"timestamp": 500, "input_length": 200, "output_length": 400, "hash_ids": [1002]}
{"timestamp": 1000, "input_length": 550, "output_length": 500, "hash_ids": [1003, 1005]}`

using the distribution from ai gateway traces.

- [ ] Model the probability distribution of the AI gateway
- [ ] Test with AI perf https://github.com/ai-dynamo/aiperf
- [ ] Extract traces from litellm
- [ ] Extract traces from opengatellm

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.