google-gemini / google-gemini/gemini-cli
perf(fileDiscovery): O(n*m) ignore filtering without subtree pruning causes multi-second delays on large repos
- Dominant language
- TypeScript
- Stars
- 107k
- Forks
- 14.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 45
Description
### What happened?
In `packages/core/src/services/fileDiscoveryService.ts` (main @ 812f7a2bc), the `filterFilesWithReport` and `getIgnoredPaths` methods exhibit O(n*m) and O(n^2) performance characteristics that degrade severely on large repositories (e.g., 50k+ files).
`fileDiscoveryService.ts` lines ~80-200:
```ts
async getIgnoredPaths(options: FilterFilesOptions = {}): Promise {
const ignoredPaths: string[] = [];
const walk = async (currentDir: string) => {
let dirEntries: fs.Dirent[];
dirEntries = await fs.promises.readdir(currentDir, { withFileTypes: true });
for (const entry of dirEntries) {
const fullPath = path.join(currentDir, entry.name);
// For each file, check against combined ignore filters
if (this.combinedIgnoreFilter.ignores(fullPath)) { // O(m) per file, where m = number of patterns
ignoredPaths.push(fullPath);
}
if (entry.isDirectory()) {
await walk(fullPath); // Recursive, no caching of ignore results for subtrees
}
}
};
await walk(this.projectRoot);
return ignoredPaths;
}
// In filterFilesWithReport:
filterFilesWithReport(relativePaths: string[], options: FilterFilesOptions): { filteredPaths: string[], ignoredCount: number } {
const filteredPaths: string[] = [];
for (const relativePath of relativePaths) {
if (this.shouldIgnore(relativePath, options)) { // O(m) per file
ignoredCount++;
} else {
filteredPaths.push(relativePath);
}
}
// ...
}
```
Issues:
1. **No subtree pruning**: `getIgnoredPaths` walks *every* directory even if the parent directory is ignored (e.g., `node_modules/` contains 30k files, all ignored, but each subdirectory is still traversed and each file checked against all patterns)
2. **No pattern caching**: `combinedIgnoreFilter.ignores()` re-evaluates all ignore patterns (`.gitignore` + `.geminiignore` + custom) for every file, even though many patterns are directory-specific and could be cached per directory
3. **Repeated realpath resolution**: `shouldIgnore` may call `resolveToRealPath` per file without caching, causing expensive stat calls on large repos
4. **Synchronous re-read of ignore files**: `IgnoreFileParser` re-reads `.gitignore`/`.geminiignore` on construction, but `FileDiscoveryService` reconstructs filters unnecessarily on option changes rather than caching
On a repository with 100k files and 200 ignore patterns, this results in 20M pattern evaluations, causing 5-10 second delays on `glob` and `read-file` tool calls that should be <500ms.
### What did you expect to happen?
- Prune ignored directory subtrees: if `isDirectoryIgnored(dir)` is true (using `ignore` library's directory matching), skip walking its children entirely
- Cache `shouldIgnore` results per file path
- Cache `resolveToRealPath` results
- Pre-compile ignore patterns into a trie or use `ignore` library's built-in optimization for directory patterns
- Lazily load and cache ignore file contents, only re-reading when files change (via mtime check)
### Client information
- Source-level finding verified against upstream `main` at commit `812f7a2bc`
- Files: `packages/core/src/services/fileDiscoveryService.ts`, `packages/core/src/utils/ignoreFileParser.ts`
- Affects all platforms, especially large monorepos
### Login information
Not applicable.
### Anything else we need to know?
Sources:
- https://github.com/google-gemini/gemini-cli/blob/812f7a2bc/packages/core/src/services/fileDiscoveryService.ts
- Repro: `time gemini --prompt "list files in src"` on a repo with `node_modules` (80k files) and 150 `.gitignore` patterns — observe 4-6s delay vs expected <1s
- Related: #28915 fixed symlink handling but did not address performance
Searched existing issues for "fileDiscovery performance", "ignore performance", "slow glob" — no open duplicate found.
Contributor guide
Research direction
Start by reading packages/core/src/services/fileDiscoveryService.ts, especially getIgnoredPaths and filterFilesWithReport, then inspect packages/core/src/utils/ignoreFileParser.ts. Reproduce the reported delay with the provided gemini prompt on a large repository and measure the current behavior. Done means large-repository file discovery avoids unnecessary ignore work and meets the stated sub-second target without changing filtering results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100