aboutcode-org / aboutcode-org/scancode-toolkit

Add intelligent file-level result caching system for faster repeated scans and CI/CD workflows

Aperta
#4,801 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
new feature
Lingua principale
Python
Stelle
2.6k
Fork
791
Merge medio
1g 12h
PR unite (30g)
5

Descrizione

## Description

Currently, ScanCode performs full scans on every run, even when files haven't changed or when running the same scan multiple times in CI/CD pipelines. This creates unnecessarily long scan times and wastes CI resources, especially for medium-to-large codebases.

### Problem Statement

1. **No incremental scanning**: Every scan processes all files, even if only a few changed
2. **No result caching**: Identical files are rescanned in subsequent runs
3. **CI/CD inefficiency**: GitHub Actions, GitLab CI, and Jenkins jobs waste time rescanning unchanged code
4. **Poor developer experience**: Local development scans take too long for iterative work

### Proposed Solution

Implement a file-level result caching system with the following features:

#### 1. Content-based Cache System
- Cache scan results per file using content hash (SHA256 of file content + ScanCode version + enabled scan options)
- Store cached results in `~/.cache/scancode/` or custom `--cache-dir` location
- Automatic cache invalidation when ScanCode version or scan options change

#### 2. CLI Integration
Add new command-line options:

--cache # Enable result caching (default: disabled)
--cache-dir PATH # Specify custom cache directory
--force-reindex # Force full rescan ignoring cache

#### 3. Smart Cache Management
- Automatic cleanup of stale cache entries
- Configurable cache size limits
- Cache statistics reporting (hit rate, time saved)

#### 4. CI/CD Optimization
- Cache survives between CI runs when using GitHub Actions cache or similar
- Show performance metrics in scan output (e.g., "Cache hits: 450/500 files, saved 5.2 minutes")

### Expected Benefits

- **40-70% faster scans** on repeated runs with few file changes
- **Reduced CI/CD costs** through shorter job times
- **Better developer experience** for local iterative scanning
- **Backward compatible**: Cache disabled by default, no breaking changes

### Implementation Notes

- Use existing `scancode.resource` and `scancode.api` infrastructure
- Follow similar pattern to existing license index caching in `licensedcode/cache.py`
- Store cached results as JSON per-file in organized directory structure
- Add comprehensive tests for cache hit/miss scenarios and invalidation
- Update documentation with caching best practices

### Use Cases

1. **PR workflows**: Scan only changed files in pull requests
2. **Incremental CI**: Faster builds when only a few files change
3. **Local development**: Quick rescans while fixing license issues
4. **Monorepo support**: Cache results across multiple scans of shared dependencies

I'm interested in implementing this as part of my GSoC 2026 contribution and would appreciate feedback on this approach.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.