openwall / openwall/john

--restore of mmap'ed wordlist should seek to offset, not skip lines

Open
#4,906 3 comments 0 reactions 1 assignee View on GitHub

@magnumripper is already working on this.

Since Nov 30, 2021.

enhancement regression
Dominant language
C
Stars
13.6k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

The way I implemented restore for wordlist mode in core, it'd store file offset in .rec file and then seek to file offset on --restore. This is broken in jumbo for mmap'ed wordlists (a feature that wasn't in core, but nevertheless a regression since mmap is enabled by default), where only line number would be stored and restoring would then (effectively) read the portion of file until the restore point is reached. For a multi-gigabyte wordlist on HDD and not currently cached in RAM, restoring a session can take ages. Even with a wordlist already in RAM, it can take longer than necessary.

Is there any reason for not seeking to offset in memory like we did in file? I think we should implement this if there's no good reason not to.

Meanwhile, we should workaround this issue by lowering the default WordlistMemoryMapMaxSize from the current 1 TB to e.g. 1 GB. This could be a good idea for another reason as well - in my experience, the Linux system feels sluggish when working with a memory-mapped wordlist of size comparable to the machine's RAM size. My guess is the kernel is more likely to discard code pages of programs, libraries, etc. in favor of other mmap'ed files than it would in favor of caching of files being read without mmap involved. (This might vary by kernel version, though.)

Setting milestone at least for us to get the workaround in. Actual code changes can wait until later (or even not be implemented at all if we're fine with keeping the file size limit for mmap low).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.