Proposal: Maintain trace-analysis quality and compatibility
- Dominant language
- TypeScript
- Stars
- 2.8k
- Forks
- 385
- Avg merge
- 5d 11m
- Merged PRs (30d)
- 34
Description
Would you be open to treating trace-analysis quality and stored-result compatibility as a maintained contract? After [#1354](), [#1376](), and [#1394](), I think the next useful step is to measure the complete analysis path rather than add another dimension.
`TraceAnalysis` is currently unversioned, older stored output is normalized by `ensureTraceAnalysisDimensions`, the registry has eight dimensions, and the tests use one trace-analysis JSON fixture.
I'd begin with a short design record to settle three decisions: keep additive-only normalization or introduce an explicit version envelope; define the boundary for a synthetic/redacted labeled corpus with per-dimension precision, recall, and F1; and choose a report-only release scorecard or a merge-blocking regression threshold.
If that direction is useful, I'd split it into three reviewable stages: schema and stored-result replay; a pilot corpus and scorecard; then an `openlit-*` release with two release cycles of follow-up for drift and false positives.
This would not add another trace-analysis dimension or expand openlit/openlit#1366's offline-evaluation configuration surface. It would stay scoped to result quality, compatibility, and the agreed release threshold.
If you want this owned as a release-spanning lane, I can start with the design record and keep the first implementation slice to schema/replay only.
Contributor guide
Research direction
Start by reading the TraceAnalysis shape, ensureTraceAnalysisDimensions, the eight-dimension registry, and the existing trace-analysis JSON fixture. Document the decisions on versioning, the labeled corpus and metrics, and the release threshold before implementing the proposed schema and stored-result replay stage. Done means the scope and reviewable stages are agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100