openlit / openlit/openlit

Proposal: Maintain trace-analysis quality and compatibility

Open
#1,424 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2.8k
Forks
385
Avg merge
5d 11m
Merged PRs (30d)
34

Description

Would you be open to treating trace-analysis quality and stored-result compatibility as a maintained contract? After [#1354](), [#1376](), and [#1394](), I think the next useful step is to measure the complete analysis path rather than add another dimension.

`TraceAnalysis` is currently unversioned, older stored output is normalized by `ensureTraceAnalysisDimensions`, the registry has eight dimensions, and the tests use one trace-analysis JSON fixture.

I'd begin with a short design record to settle three decisions: keep additive-only normalization or introduce an explicit version envelope; define the boundary for a synthetic/redacted labeled corpus with per-dimension precision, recall, and F1; and choose a report-only release scorecard or a merge-blocking regression threshold.

If that direction is useful, I'd split it into three reviewable stages: schema and stored-result replay; a pilot corpus and scorecard; then an `openlit-*` release with two release cycles of follow-up for drift and false positives.

This would not add another trace-analysis dimension or expand openlit/openlit#1366's offline-evaluation configuration surface. It would stay scoped to result quality, compatibility, and the agreed release threshold.

If you want this owned as a release-spanning lane, I can start with the design record and keep the first implementation slice to schema/replay only.

Contributor guide

Open the contributing guide

Research direction

Start by reading the TraceAnalysis shape, ensureTraceAnalysisDimensions, the eight-dimension registry, and the existing trace-analysis JSON fixture. Document the decisions on versioning, the labeled corpus and metrics, and the release threshold before implementing the proposed schema and stored-result replay stage. Done means the scope and reviewable stages are agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.