agoda-com / agoda-com/AgodaAnalyzers
Add performance tests for the rules using BenchmarkDotNet
- Linguagem predominante
- C#
- Estrelas
- 25
- Forks
- 15
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
## Problem Statement
Currently, we lack visibility into the performance characteristics of our Roslyn analyzers. Without performance benchmarks, we risk introducing performance regressions that could significantly impact developer experience in IDEs and CI/CD pipelines. We need a comprehensive performance testing harness to:
- Measure analyzer performance under different conditions
- Detect performance regressions in CI
- Identify optimization opportunities
- Ensure analyzers scale well with large codebases
## Proposed Solution
### Overview
Implement a BenchmarkDotNet-based performance testing suite that measures analyzer performance across various scenarios and code complexity levels.
### Detailed Plan
#### Phase 1: Infrastructure Setup
**1.1 Create Performance Test Project Structure**
```
Agoda.Analyzers.PerformanceTests/
├── Agoda.Analyzers.PerformanceTests.csproj
├── Benchmarks/
│ ├── IndividualRuleBenchmarks.cs
│ ├── FullAnalyzerSuiteBenchmarks.cs
│ └── ScalabilityBenchmarks.cs
├── TestData/
│ ├── NoViolations/
│ │ ├── LargeCodebase/
│ │ ├── MediumCodebase/
│ │ └── SmallCodebase/
│ ├── WithViolations/
│ │ ├── AllRulesViolated/
│ │ ├── HighViolationDensity/
│ │ └── LowViolationDensity/
│ └── EdgeCases/
│ ├── DeepInheritanceHierarchy/
│ ├── LargeFiles/
│ └── ComplexExpressions/
├── Utilities/
│ ├── CodeGenerator.cs
│ ├── TestDataManager.cs
│ └── ReportGenerator.cs
└── Scripts/
├── generate-test-data.ps1
└── run-benchmarks.ps1
```
**1.2 Dependencies and Configuration**
- Add BenchmarkDotNet package
- Add Microsoft.CodeAnalysis.Testing packages
- Configure benchmark attributes and parameters
- Set up memory and CPU profiling
#### Phase 2: Test Data Generation
**2.1 No Violations Test Projects**
Create synthetic codebases that don't trigger any analyzer rules:
- **Small**: ~50 files, 5K LOC - Basic classes, methods, properties
- **Medium**: ~200 files, 25K LOC - Realistic project structure with dependencies
- **Large**: ~1000 files, 100K LOC - Enterprise-scale codebase simulation
**2.2 With Violations Test Projects**
Create codebases that intentionally trigger rules:
- **All Rules Violated**: Each rule triggered at least once per file
- **High Density**: Multiple violations per file (stress test)
- **Low Density**: Sparse violations (realistic scenario)
**2.3 Edge Case Scenarios**
- Files with 10K+ lines of code
- Deep inheritance hierarchies (20+ levels)
- Complex LINQ expressions and nested lambdas
- Large number of using statements and namespace nesting
- Auto-generated code patterns
**2.4 Code Generation Utility**
```csharp
public class CodeGenerator
{
public void GenerateNoViolationCodebase(int fileCount, int averageLinesPerFile);
public void GenerateViolationCodebase(RuleSet rules, ViolationDensity density);
public void GenerateEdgeCaseScenarios();
}
```
#### Phase 3: Benchmark Implementation
**3.1 Individual Rule Benchmarks**
```csharp
[MemoryDiagnoser]
[SimpleJob(RuntimeMoniker.Net60)]
[SimpleJob(RuntimeMoniker.Net80)]
public class IndividualRuleBenchmarks
{
[Params("Small", "Medium", "Large")]
public string CodebaseSize { get; set; }
[Params("NoViolations", "WithViolations")]
public string ViolationType { get; set; }
[Benchmark]
[ArgumentsSource(nameof(GetAllRules))]
public async Task BenchmarkRule(string ruleName)
{
// Benchmark individual rule performance
}
}
```
**3.2 Full Analyzer Suite Benchmarks**
- Measure complete analyzer suite performance
- Test with/without violations
- Different codebase sizes
- Memory usage tracking
**3.3 Scalability Benchmarks**
- Linear scaling tests (1x, 2x, 4x, 8x code size)
- File count vs file size impact analysis
- Concurrent analysis performance
#### Phase 4: Performance Metrics and Reporting
**4.1 Key Metrics to Capture**
- **Execution Time**: Mean, median, P95, P99
- **Memory Usage**: Peak memory, GC pressure
- **Throughput**: Files analyzed per second
- **Scaling Characteristics**: Performance vs codebase size
**4.2 Baseline and Regression Detection**
- Store baseline performance metrics
- Automated comparison with previous runs
- Configurable performance regression thresholds
- Performance trend analysis
**4.3 Reporting Format**
- HTML reports with charts and graphs
- JSON output for CI integration
- Markdown summaries for PR comments
- Historical performance tracking
#### Phase 5: GitHub Actions Integration
**5.1 Performance Test Workflow**
```yaml
name: Performance Tests
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main ]
schedule:
# Run nightly performance tests
- cron: '0 2 * * *'
jobs:
performance-tests:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: |
6.0.x
8.0.x
- name: Restore dependencies
run: dotnet restore
- name: Generate test data
run: ./Scripts/generate-test-data.ps1
- name: Run benchmarks
run: dotnet run -c Release --project Agoda.Analyzers.PerformanceTests
- name: Upload benchmark results
uses: actions/upload-artifact@v4
with:
name: benchmark-results
path: BenchmarkDotNet.Artifacts/
- name: Performance regression check
run: ./Scripts/check-performance-regression.ps1
- name: Comment PR with results
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
// Post performance summary to PR
```
**5.2 Performance Monitoring Workflow**
```yaml
name: Performance Monitoring
on:
schedule:
- cron: '0 6 * * 1' # Weekly detailed analysis
jobs:
detailed-performance-analysis:
runs-on: ubuntu-latest
steps:
- name: Extended benchmark suite
run: # Run comprehensive benchmarks
- name: Generate performance report
run: # Create detailed HTML report
- name: Update performance dashboard
run: # Update GitHub Pages dashboard
```
#### Phase 6: CI/CD Integration Details
**6.1 Performance Gates**
- Fail CI if performance degrades by >20%
- Warning if performance degrades by >10%
- Different thresholds for different rule categories
**6.2 Conditional Execution**
```yaml
- name: Check if performance tests needed
id: check-changes
run: |
# Only run expensive tests if analyzer code changed
if git diff --name-only HEAD~1 | grep -E "(Analyzers/|Rules/)"; then
echo "run-perf-tests=true" >> $GITHUB_OUTPUT
fi
- name: Run performance tests
if: steps.check-changes.outputs.run-perf-tests == 'true'
run: dotnet run -c Release --project Agoda.Analyzers.PerformanceTests
```
**6.3 Results Storage and Tracking**
- Store results in GitHub artifacts
- Optional: Store in external system (Azure Storage, etc.)
- Performance history tracking
- Automated performance regression alerts
### Implementation Checklist
#### Infrastructure
- [ ] Create performance test project
- [ ] Add BenchmarkDotNet dependencies
- [ ] Set up project structure
- [ ] Configure benchmark parameters
#### Test Data
- [ ] Implement code generation utilities
- [ ] Generate no-violation test codebases
- [ ] Generate with-violation test codebases
- [ ] Create edge case scenarios
- [ ] Validate test data quality
#### Benchmarks
- [ ] Implement individual rule benchmarks
- [ ] Implement full suite benchmarks
- [ ] Implement scalability benchmarks
- [ ] Add memory profiling
- [ ] Configure multiple runtime targets
#### Reporting
- [ ] Implement baseline storage
- [ ] Create performance regression detection
- [ ] Generate HTML reports
- [ ] Create markdown summaries
#### CI/CD
- [ ] Create performance test workflow
- [ ] Implement conditional execution
- [ ] Add performance gates
- [ ] Set up artifact storage
- [ ] Configure PR commenting
#### Documentation
- [ ] Add README for performance tests
- [ ] Document benchmark methodology
- [ ] Create troubleshooting guide
- [ ] Add performance optimization guidelines
### Performance Optimization Focus Areas
Based on common Roslyn analyzer performance issues, pay special attention to:
1. **Node Filtering**: Ensure analyzers only register for relevant syntax nodes
2. **Symbol Resolution**: Minimize expensive symbol lookups
3. **Tree Traversal**: Avoid unnecessary full tree walks
4. **Memory Allocations**: Minimize allocations in hot paths
5. **Diagnostic Creation**: Efficient diagnostic reporting
### Success Criteria
- [ ] All existing rules have performance baselines established
- [ ] Performance regression detection is working in CI
- [ ] Performance improves or remains stable for new rule additions
- [ ] Documentation and tooling enable easy performance analysis
- [ ] Team can identify and fix performance issues quickly
### Timeline
- **Week 1-2**: Infrastructure setup and project structure
- **Week 3-4**: Test data generation and validation
- **Week 5-6**: Benchmark implementation
- **Week 7**: GitHub Actions integration
- **Week 8**: Documentation and refinement
### Future Enhancements
- Integration with continuous performance monitoring
- Performance comparison across .NET versions
- Automated performance optimization suggestions
- Integration with IDE performance profiling tools
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.