KaykCaputo / KaykCaputo/oracletrace

[Feature]: Retrieve baseline automatically from branch or commit

Open
#72 2 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted important
Dominant language
Python
Stars
22
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Summary

Add support for automatically retrieving a baseline trace from a Git branch or a specific Git commit.

Proposed flags:

```bash
oracletrace app.py --get-baseline-from-branch main

oracletrace app.py --get-baseline-from-commit a1b2c3d
```

## Problem

Today, performance comparisons require manually generating, storing, and providing baseline JSON files.

Typical CI workflows require additional scripting to:

1. Checkout a reference branch or commit.
2. Generate a baseline trace.
3. Store the generated JSON artifact.
4. Restore it later for comparison.

This introduces operational overhead and makes regression detection harder to adopt.

## Proposed Solution

Allow OracleTrace to automatically obtain a baseline from Git history.

### Branch mode

```bash
oracletrace app.py --get-baseline-from-branch main
```

Workflow:

1. Resolve the latest commit from the specified branch.
2. Locate a previously generated OracleTrace baseline associated with that commit.
3. Use it as the comparison baseline.

### Commit mode

```bash
oracletrace app.py --get-baseline-from-commit a1b2c3d
```

Workflow:

1. Resolve the specified commit.
2. Locate a baseline associated with that commit.
3. Use it for comparison.

Internally this could integrate with:

- Local cache directory
- CI artifacts
- Future remote baseline storage backends

## Example

Current workflow:

```bash
oracletrace app.py --json baseline.json

oracletrace app.py \
--json current.json \
--compare baseline.json
```

Desired workflow:

```bash
oracletrace app.py \
--get-baseline-from-branch main \
--fail-on-regression \
--threshold 10
```

or

```bash
oracletrace app.py \
--get-baseline-from-commit a1b2c3d \
--fail-on-regression \
--threshold 10
```

## Use Case

### Pull Request validation

A CI pipeline could automatically compare a feature branch against the latest baseline generated on `main`.

```bash
oracletrace app.py \
--get-baseline-from-branch main \
--fail-on-regression
```

### Historical regression investigation

Developers could compare performance against a known-good commit.

```bash
oracletrace app.py \
--get-baseline-from-commit a1b2c3d
```

## Alternatives Considered

- Continue managing baseline files manually.
- Store baseline paths in CI configuration.
- Use external scripts to fetch historical baselines before running OracleTrace.

## Open Questions

- Where should baselines be stored and indexed?
- Local cache
- GitHub Actions artifacts
- Dedicated remote storage
- Should OracleTrace generate the baseline automatically if one does not exist?
- How should branch names map to baseline artifacts?

## Additional Context

This feature would significantly improve CI/CD integration by reducing manual baseline management and enabling Git-native performance comparisons.

## Checklist

- [x] I searched existing issues before opening this request
- [x] I described the problem and why this feature is useful
- [x] I provided enough detail for implementation discussion

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.