Review default incremental charsets provided with John

The ascii.chr file that's used by default for incremental mode hasn't been change since at least 2013 (apart from an accidental change that was subsequently reverted), and doesn't reflect many of the common patterns that are seen in many passwords on newer systems.

In my experience, the majority of systems now enforce some kind of password complexity rules (for better or worse), and the most common way that users adapt to these rules is usually by sticking a capital letter and a number into their password (usually at the start and end respectively). However, based on a sample of the first 4 million candidates, generated from ascii.chr, the vast majority of the generated candidates don't include any uppercase characters:

Character sets
loweralpha: 1769406 (45.97%)
loweralphanum: 1740805 (45.23%)
numeric: 311967 (8.11%)
loweralphaspecial: 11281 (0.29%)
loweralphaspecialnum: 10834 (0.28%)
specialnum: 1677 (0.04%)
mixedalphanum: 723 (0.02%)
upperalphanum: 639 (0.02%)
mixedalpha: 260 (0.01%)
mixedalphaspecial: 105 (0.0%)
upperalphaspecialnum: 61 (0.0%)
upperalpha: 27 (0.0%)
special: 23 (0.0%)
mixedalphaspecialnum: 21 (0.0%)

And the majority of the passwords are also less than 8 characters long, which is a common minimum length:

Password length (length ordered)
1 = 94 (0.0%)
2 = 343 (0.01%)
3 = 2324 (0.06%)
4 = 26429 (0.66%)
5 = 443440 (11.09%)
6 = 2450977 (61.27%)
7 = 823543 (20.59%)
8 = 250000 (6.25%)
9 = 2592 (0.06%)
10 = 256 (0.01%)

This means that the out-the-box incremental mode is very ineffective on any system that has password complexity.

As a test, I ran the default ascii.chr incremental modes for 30 minutes against a recent Active Directory dump containing ~56k unique hashes - it cracked 446 of them.

By comparison, generating a charset from just the ~1,100 mixed-alpha-numeric passwords in the NCSC Top 100K Common Passwords cracked 3,560 hashes in the same time. And a charset generated from ~20,000 previously broken NT hashes from other unrelated Active Directory domains cracked 7,001 hashes.

I'm not saying that 1,100 words from a common password list is good way to build a character set - but although crude it seemed to be actually pretty effective.

My usage of John is heavily focused on English-speaking enterprise environments (and often Active Directory), which have different password policies and patterns from other systems, so I appreciate that this is not representative of how other people are using it. Perhaps the current incremental charsets are effective for most users, and the way I use John just makes me a bit of an outlier. And if that's the case, I'm quite happy generating my own.

But I think that it's perhaps worth revisiting the charsets included with John and considering:

• Would a mixed-alpha-numeric charset be a useful addition?
• Is the current ascii.chr still a good default in 2022?
• More broadly, should the charsets be reviewed and updated?

Start by reviewing run/ascii.chr and the incremental mode that consumes it, then compare candidate charset changes against the password-pattern and cracking results described in the issue. Done means reaching agreement on whether the defaults should change and, if so, updating the included charsets with supporting validation.