JamieMason / JamieMason/syncpack

feat: prune ignored directories during file discovery

Open
#338 2 comments 0 reactions 0 assignees View on GitHub
Priority: Medium Status: Blocked Type: Upstream
Dominant language
Rust
Stars
2.1k
Forks
72
PR merge metrics
No merged PRs in 30d

Description

### Description

Syncpack's file discovery walks the entire workspace tree, visiting every directory except `node_modules` and `.git`. It does not prune directories excluded by `.gitignore`, nor does it limit traversal to the directories implied by the workspace globs, even when explicit `--source` flags are passed.

In large monorepos this can dominate total runtime. In our workspace (~630 projects, pnpm workspaces, 120 package.json files), with a 1.4M-file jest cache directory at the workspace root (gitignored): syncpack lint took ~63 seconds. After removing that directory: ~4.5 seconds. In a minimal repro (3 packages), adding 40,000 gitignored junk files took syncpack lint from 0.05s to ~13.5s.

Additionally, on corporate machines like mine, endpoint protection (Microsoft Defender) intercepts the walker's per-file accesses, at ~0.3ms per not-yet-cached file. When there are more files on the walk, this adds up over time, making the walk even slower. Since build/test caches regenerate files constantly, the cost recurs on every run.

Reproduced on 14.3.0 and 15.3.2. The 15.2.0 fix for #334 corrected which files appear in results, but the walk itself still visits everything. Syncpack seems the right place to fix this because no CLI flag can currently avoid the traversal (`--source` filters results, not the walk).

### Suggested Solution

No CLI or config surface change strictly needed, ideally discovery just gets faster.

- Prune traversal using `.gitignore` (e.g. the ignore crate's WalkBuilder, which handles nested gitignores natively), the same way ripgrep does
- Only descend into directories that can match the workspace globs / source patterns (e.g. `packages: ["apps/*", "libs/*"]` never requires entering `./.jestcache/**` or `./dist/**`).

If implicit gitignore behavior is undesirable, an opt-in config would also solve it:

```json
{
"ignorePaths": [".jestcache/**", "dist/**", "coverage/**"]
}
```

Expected outcome: `syncpack lint` output unchanged, but runtime proportional to the number of candidate package files rather than total files in the repo.

### Optional comments

_No response_

### Code of Conduct

- [x] I agree to follow the [Code of Conduct](https://github.com/JamieMason/syncpack/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start at syncpack's file-discovery entry point and the traversal used by `syncpack lint`, then reproduce the slowdown with a large gitignored directory or workspace globs such as `apps/*` and `libs/*`. Compare behavior with explicit `--source` flags and verify that lint output is unchanged while traversal avoids irrelevant directories and runtime improves.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, rust
Domain
cli, performance, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.