invenia / invenia/Checkpoints.jl
`index_files(path::S3Path)` is very slow on large directories
- Dominant language
- Julia
- Stars
- 12
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
It is because `isfile` being run on each path returned by `collect(walkpath)` [on this line](https://github.com/invenia/Checkpoints.jl/blob/main/src/indexing.jl#L156).
One possible workaround would be to filter out paths which end in `/`. For a 3000 file `path` the difference is 0.8s vs 60s using `isfile`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at src/indexing.jl line 156 and inspect how index_files(path::S3Path) processes the paths returned by collect(walkpath). Reproduce the reported comparison on a large directory, then verify that indexing avoids the slow per-path isfile calls while still selecting the intended files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100