--mem-file-size should possibly apply per-invocation or per-file, not per-process
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 13.6k
- Forks
- 2.6k
- PR merge metrics
- No merged PRs in 30d
Description
When running with a huge wordlist and a high --fork count, and with otherwise default settings, startup takes ages and a lot of RAM is consumed. This appears to be because our default for --mem-file-size on 64-bit is 2 GB and we apply this limit per-process. For example, with a 60 GB wordlist and 32 forks, the entire wordlist would currently be read into RAM on startup, right? Worse, on a HDD the 32 processes would compete for disk seeks back and forth, right?
To remedy this, I think the limit should apply per-invocation or per-file, not per-process. For example, if we split a job across several machines with --node, it could be fine for each machine to have its own full limit on loading its portion of wordlist into memory. However, we shouldn't want a full limit per each virtual node within each physical machine's node numbers' range created there with --fork. If that's unreasonably complicated to implement and/or document, which I think it may be, then perhaps we should simply apply the limit per-file.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the --mem-file-size, --fork, and --node command-line handling and how wordlists are loaded per process. Reproduce startup with a large wordlist and multiple forks, then determine and document whether the limit applies per invocation, per file, or per process. Done means the selected scope is implemented and startup memory and disk behavior are covered by relevant tests or reproducible checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- cli, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100