openwall / openwall/john

--mem-file-size should possibly apply per-invocation or per-file, not per-process

Open
#4,907 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement RFC / discussion
Dominant language
C
Stars
13.6k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

When running with a huge wordlist and a high --fork count, and with otherwise default settings, startup takes ages and a lot of RAM is consumed. This appears to be because our default for --mem-file-size on 64-bit is 2 GB and we apply this limit per-process. For example, with a 60 GB wordlist and 32 forks, the entire wordlist would currently be read into RAM on startup, right? Worse, on a HDD the 32 processes would compete for disk seeks back and forth, right?

To remedy this, I think the limit should apply per-invocation or per-file, not per-process. For example, if we split a job across several machines with --node, it could be fine for each machine to have its own full limit on loading its portion of wordlist into memory. However, we shouldn't want a full limit per each virtual node within each physical machine's node numbers' range created there with --fork. If that's unreasonably complicated to implement and/or document, which I think it may be, then perhaps we should simply apply the limit per-file.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the --mem-file-size, --fork, and --node command-line handling and how wordlists are loaded per process. Reproduce startup with a large wordlist and multiple forks, then determine and document whether the limit applies per invocation, per file, or per process. Done means the selected scope is implemented and startup memory and disk behavior are covered by relevant tests or reproducible checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
cli, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.