Bug: Runaway memory usage when run on large directories
- Dominant language
- Rust
- Stars
- 3.1k
- Forks
- 96
- PR merge metrics
- No merged PRs in 30d
Description
Hey @imsnif ! Another awesome application! Hopefully I can help out with some of those easier issues you've added to the list sometime soon!
I've run into a bit of an issue in running diskonaut on my server through: after around an hour of indexing files, diskonaut was using 22GB of RAM! This pushed a bunch of system processes to swap and brought down a couple of services I had running.
I'll have to take a peek at the code, but I'm assuming this is from tracking the name and information of every file encountered during a scan. If that's the case, then perhaps diskonaut could intelligently decided when RAM usage is getting a bit out of control and collapse sub-trees into a single total size. That would mean that, if you wanted to dive into that sub-tree, you might need to re-index it, but it's just an idea. Having the current directory fully indexed would be good to keep zooming in and out quickly, but I think it's acceptable to re-index sub-directories when you enter them.
As I understand, some other disk-usage applications (like `baobab` from [GNOME](https://github.com/GNOME/baobab)) don't even bother tracking individual files. I think it's good that diskonaut does allow you to view those (I'm absolutely in love with the zoom feature!!), but maybe it shouldn't store information for every file _all_ of the time.
Thanks again for the awesome app!
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reproducing the scan on a large directory and inspect how indexing retains file and directory information. Done should mean substantially lower memory usage during long scans without losing the disk-usage navigation behavior described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100