aws-samples / aws-samples/foundation-model-benchmarking-tool
Emit live metrics
- Dominant language
- Jupyter Notebook
- Stars
- 256
- Forks
- 44
- PR merge metrics
- No merged PRs in 30d
Description
Emit live metrics so that they can be monitored through Grafana via live dashboard. More information to come on this issue but the goal here is to provide full flexibility to the user to be able to view metrics in ways that best suits the needs of their business and technological goals.
[TBD] --> Some sort of an analytics pipeline sending and emitting live results for different model configurations, their results and different metrics _**based on the needs of the user**_.
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by mapping the existing benchmarking and metrics flow, then clarify the analytics pipeline and supported model configurations; done means live results and metrics can be emitted and viewed through a Grafana dashboard.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100