[Discuss] <Large Language Model (LLM) systems Metrics>
Open
question
- Dominant language
- Java
- Stars
- 7.4k
- Forks
- 1.3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 32
Description
### Discuss
monitoring llm metrics and more.
https://medium.com/data-science-at-microsoft/evaluating-llm-systems-metrics-challenges-and-best-practices-664ac25be7e5
Contributor guide
Research direction
Start by reading the linked article and the repository's existing monitoring and metrics architecture; the issue names no files, tests, or entry points. Before implementation, define which LLM metrics are needed, where they should be collected, and what validation will demonstrate that the monitoring is complete.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100