invenia / invenia/Checkpoints.jl

`index_files(path::S3Path)` is very slow on large directories

Open
#35 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
12
Forks
4
PR merge metrics
No merged PRs in 30d

Description

It is because `isfile` being run on each path returned by `collect(walkpath)` [on this line](https://github.com/invenia/Checkpoints.jl/blob/main/src/indexing.jl#L156).

One possible workaround would be to filter out paths which end in `/`. For a 3000 file `path` the difference is 0.8s vs 60s using `isfile`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at src/indexing.jl line 156 and inspect how index_files(path::S3Path) processes the paths returned by collect(walkpath). Reproduce the reported comparison on a large directory, then verify that indexing avoids the slow per-path isfile calls while still selecting the intended files.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.