demml / demml/opsml

Embedding metrics

Open
#228 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
40
Forks
3
PR merge metrics
No merged PRs in 30d

Description

LLM as a judge is important, but it's also imperative to provide some form of statistical insight into how data is changing over time when it comes to inputs and outputs.

Use case:

A user would like to access changes in the distributions of the their input and outputs over time

Approach:

- Generate embeddings for input and output
- Average the embedding? - tbd, research this
- Accumulate results over time:
- similarities
- Skew
- Mean,
- median
- mode
- distribution plots, etc.

Contributor guide

Open the contributing guide

Research direction

No file, test, or entry point is named. Start by researching how input and output embeddings should be generated and aggregated over time, then define the required similarity statistics, distribution measures, and plots; done criteria are not specified in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
analytics, machine-learning, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.