openwall / openwall/john

parallel expansion in preprocessor: \p0 does not work if previous range has \p modifier too

Open
#4,117 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C
Stars
13.6k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

It is not possible to use \p0 to chain parallel expansion of several ranges together. \p or \pN should be used.

Chaining of ranges is useful to merge 2 lines into 1 (current ReplaceSpecial2Special in korelogic.conf uses such technique). Merging lines is cool because it gives an easy to use gadgets that can be combined with other rules inline (e.g. [ab] + [cd] gives 4 rules in 1 lines, but ab and cd are longer). \p0 is important because it makes gadgets independent of context, i.e. gadget may be pasted before or after other gadget without modifications.

Preps:

$ echo X > X

Let's merge Az"abcd" and Az"1234".

$ ./JohnTheRipper/run/john --wordlist=X --stdout --rules=':Az"[a1]\p1[b2]\p1[c3]\p1[d4]"'
Using default input encoding: UTF-8
Xabcd
X1234
2p 0:00:00:00 100.00% (2019-10-12 00:16) 20.00p/s X1234

But with \p0 it does not work:

$ ./JohnTheRipper/run/john --wordlist=X --rules=':Az"[a1]\p0[b2]\p0[c3]\p0[d4]"' --stdout
Using default input encoding: UTF-8
Xabcd
X12cd
2p 0:00:00:00 100.00% (2019-10-12 00:06) 25.00p/s X12cd

[a1] is expanded as regular range. [b2] references [a1]. Next references fail. Same happens with \p1[b2] and \p0[c3].

John proper has the same limitation. I modified the config file.

$ cat john-1.9.0/run/john.conf
...
[List.Rules:Wordlist]
Az"[a1]\p0[b2]\p0[c3]\p0[d4]"
...

$ ./john-1.9.0/run/john --wordlist=X --rules --stdout
Press 'q' or Ctrl-C to abort, almost any other key for status
Xabcd
X12cd
2p 0:00:00:00 100% 50.00p/s X12cd

jumbo is quite fresh (has opencl; openmp is disabled):

Version: 1.9.0-jumbo-1+bleeding-ea33667eb 2019-09-23 12:43:34 +0200
Build: linux-gnu 64-bit x86_64 AVX2 AC

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the behavior with ./john --wordlist=X --rules --stdout and the rule shown in the issue, then inspect the rule-processing entry points used by John. Compare the \p1 and \p0 cases, including the List.Rules:Wordlist example in john.conf and the ReplaceSpecial2Special usage in korelogic.conf. Done means \p0 can chain all four ranges and produce the expected combinations.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.