githubnext / githubnext/gh-aw-trial-hono

Daily Perf Improver - Historical Performance Tracking CI Integration

Open
#41 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary

Integrates automated historical performance tracking into the CI pipeline, completing Medium Priority #4 from the performance status report. This enables long-term trend analysis, regression detection across releases, and data-driven optimization decisions.

## Goal and Rationale

**Performance Engineering Goal:** Build historical performance tracking infrastructure for trend analysis and regression detection

**Why it matters:**
- Enables visualization of performance trends over weeks/months/releases
- Provides early warning system for performance regressions
- Documents performance milestones for release planning
- Supports data-driven optimization decisions with historical context

## Approach

The historical tracking tools (`track-performance.ts`, `analyze-performance.ts`, `visualize-performance.ts`) already existed but were not integrated with CI. This PR completes the integration by:

1. **CI Automation** - Added `http-benchmark-on-main` job that runs on every main branch push
2. **Data Persistence** - Commits performance history to dedicated `perf-history` branch
3. **Developer Tools** - Added npm scripts for easy access to tracking tools
4. **Documentation** - Comprehensive usage guide with workflows and best practices

## Impact

### CI Integration

**Main Branch Automation:**
- Every commit to `main` automatically runs HTTP benchmarks
- Saves results with git metadata (commit SHA, branch, message)
- Commits to `perf-history` branch for permanent storage
- Zero overhead on PR workflow (only runs on main)

**Data Structure:**
```jsonl
{"timestamp":"2025-10-17T10:30:00Z","git":{"branch":"main","commit":"abc1234"},"results":{...}}
```

### Developer Experience

**New Package Scripts:**
```bash
bun run perf:track # Save benchmark results to history
bun run perf:analyze # Analyze performance trends
bun run perf:visualize # Generate ASCII charts
bun run perf:html # Create HTML visualizations
```

**Example Workflow - Trend Analysis:**
```bash
# View last 30 days of performance data
bun run perf:analyze --since=30d

# Visualize trends
bun run perf:visualize --since=30d --metric=overall

# Generate HTML report for release notes
bun run perf:html --output=release-performance.html
```

## Implementation Details

### CI Workflow Changes

Added new job in `.github/workflows/ci.yml`:

```yaml
http-benchmark-on-main:
name: 'HTTP Benchmark & Historical Tracking'
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
permissions:
contents: write # Needed to commit historical data
steps:
- Run HTTP benchmark
- Track historical performance
- Commit to perf-history branch
```

**Key Features:**
- Only runs on `main` branch (not PRs)
- Uses `fetch-depth: 0` for full git history
- Auto-creates `perf-history` branch on first run
- Uses `[skip ci]` in commit messages to prevent infinite loops

### Package.json Scripts

```json
{
"scripts": {
"perf:track": "cd benchmarks/http-server && bun run track-performance.ts",
"perf:analyze": "cd benchmarks/http-server && bun run analyze-performance.ts",
"perf:visualize": "cd benchmarks/http-server && bun run visualize-performance.ts",
"perf:html": "cd benchmarks/http-server && bun run visualize-performance.ts --output=performance-chart.html"
}
}
```

### Documentation

**Comprehensive Guide:** `benchmarks/http-server/README-HISTORICAL-TRACKING.md` (400+ lines)

Covers:
- System overview and components
- Usage examples for all tools
- CI integration details
- Common workflows (local dev, release planning, regression investigation)
- Advanced filtering and output formats
- Troubleshooting guide
- Best practices

**Quick Reference:** Updated `benchmarks/http-server/README.md` with quick start examples

## Validation

### Testing

**Local Testing:**
```bash
# Verified tracking tools work correctly
cd benchmarks/http-server
bun run benchmark.ts
bun run track-performance.ts
bun run analyze-performance.ts
bun run visualize-performance.ts

# Confirmed data structure and git metadata capture
✅ All tools function correctly
✅ Git metadata extracted properly
✅ JSONL format validated
```

**CI Workflow:**
- ✅ Syntax validated
- ✅ Permissions configured correctly (`contents: write`)
- ✅ Branch conditions verified (`if: github.ref == 'refs/heads/main'`)
- ✅ Fetch depth set appropriately
- ✅ Skip CI marker prevents loops

### Example Analysis Output

```
📊 Summary Statistics (overall)
──────────────────────────────────────────────────
Total entries: 15
Mean: 125,432.50 req/s
Median: 126,123.00 req/s
Min: 118,234.00 req/s
Max: 131,789.00 req/s
Trend: 📈 Improving

📋 Recent Performance History
──────────────────────────────────────────────────
Date Commit Branch Value Change Message
──────────────────────────────────────────────────
2025-10-17 10:30:00 abc1234 main 131,789.30 +5.1% Optimize context
2025-10-16 15:20:00 def5678 main 125,456.20 +2.3% JSX improvements
```

## Trade-offs

**Pros:**
- **Permanent Performance History** - Never lose historical data
- **Zero PR Overhead** - Only runs on main, not PRs
- **Easy Access** - Simple npm scripts for all operations
- **Comprehensive Docs** - Clear workflows and examples
- **Minimal Storage** - ~1KB per commit, grows linearly

**Cons:**
- **Main Branch Only** - No historical tracking for feature branches (by design)
- **Requires Main Push** - First `perf-history` branch creation needs main push
- **Minimal Overhead** - Adds ~1-2 minutes to main CI (acceptable for value gained)

## Future Enhancements

Potential improvements identified but not implemented:
1. **Automatic Regression Alerts** - Notify maintainers on significant degradations
2. **Multi-Metric Dashboards** - Combined view of all performance dimensions
3. **Release Comparison Tool** - Side-by-side performance across versions
4. **Percentile Tracking** - P50/P95/P99 latency history
5. **Memory Tracking Integration** - Combine with memory profiling tools

## Related Work

- Completes **Medium Priority #4** from `.github/copilot/instructions/performance-status-report.md`
- Builds on existing HTTP benchmark infrastructure (statistical analysis, confidence intervals)
- Integrates with performance budget validation system
- Complements profiling tools and memory tracking

## Usage After Merge

**For Maintainers:**
1. Merge this PR to main
2. First main push auto-creates `perf-history` branch
3. Every subsequent main push adds performance data
4. Use `bun run perf:analyze` to review trends monthly

**For Contributors:**
```bash
# Track local optimization experiments
bun run build
cd benchmarks/http-server
bun run benchmark.ts
bun run perf:track --tag="my-optimization-experiment"
bun run perf:analyze --limit=5
```

**For Releases:**
```bash
# Tag release milestones
bun run perf:track --tag="v4.10.0-release"

# Generate release performance report
bun run perf:analyze --since=30d --format=json > release-perf.json
bun run perf:html --since=30d
```

---

🤖 Generated with Claude Code

Co-Authored-By: Claude

> AI generated by [Daily Perf Improver](https://github.com/githubnext/gh-aw-trial-hono/actions/runs/18583834816)
---
> [!NOTE]
> This was originally intended as a pull request, but the git push operation failed.
>
> **Workflow Run:** [View run details and download patch artifact](https://github.com/githubnext/gh-aw-trial-hono/actions/runs/18583834816)
>
> The patch file is available as an artifact (`aw.patch`) in the workflow run linked above.
To apply the patch locally:
```sh
# Download the artifact from the workflow run https://github.com/githubnext/gh-aw-trial-hono/actions/runs/18583834816
# (Use GitHub MCP tools if gh CLI is not available)
gh run download 18583834816 -n aw.patch
# Apply the patch
git am aw.patch
```

Show patch preview (500 of 600 lines)

```diff
From a731dc3dbbb8e2574701063caf15723d2a9c564e Mon Sep 17 00:00:00 2001
From: Daily Perf Improver
Date: Fri, 17 Oct 2025 06:00:18 +0000
Subject: [PATCH] feat: integrate historical performance tracking with CI
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Add automated performance tracking infrastructure to CI pipeline for
long-term trend analysis and regression detection.

## Changes

### CI Integration
- Add http-benchmark-on-main job that runs on every main branch push
- Automatically tracks performance data with git metadata
- Commits historical data to perf-history branch for permanent storage
- Zero-overhead on PR workflow (only runs on main)

### Package Scripts
Add convenience commands for performance tracking tools:
- bun run perf:track - Save benchmark results to history
- bun run perf:analyze - Analyze performance trends
- bun run perf:visualize - Generate ASCII charts
- bun run perf:html - Create interactive HTML visualizations

### Documentation
- Comprehensive README-HISTORICAL-TRACKING.md with complete usage guide
- Updated main benchmark README with quick start examples
- Common workflows for local development and release planning
- Troubleshooting guide and best practices

## Impact

**For Main Branch:**
Every commit automatically collects performance data enabling:
- Long-term trend visualization across releases
- Early regression detection through historical comparison
- Performance milestone tracking for release planning
- Data-driven optimization decisions

**For Developers:**
Easy access to performance history via npm scripts:
- Quick trend analysis with bun run perf:analyze
- Visual charts with bun run perf:visualize
- HTML reports for documentation with bun run perf:html

## Implementation

Historical tracking tools already exist (track-performance.ts,
analyze-performance.ts, visualize-performance.ts) but were not
integrated with CI. This PR completes th
... (truncated)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.