openwall / openwall/john

Improve rule stacking

Open
#4,014 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
C
Stars
13.6k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

Current rule stacking actually runs the preprocessor as well as the rule engine twice, which can't help performance. I'm thinking we could instead concatenate the two rules before PP and only run things once although that will involve less trivial things for throwing things around (eg. if the second rule has reject-flags they must be moved to just after the first rule's reject flags (if any) and/or remove dupes of such.

Example, rule1 ^[A-Z] and rule2 Az"[0-9][0-9]"

Current flow (first entry from PP from respective rule):

"word" -> apply ^A -> "Aword" -> apply Az"00" -> "Aword00"

New flow, single rule, PP ^[A-Z] Az"[0-9][0-9]"

"word" -> apply ^A Az"00" -> "Aword00"

That was easy. Now something less trivial:

Rule1 -p -c (?a 2 (?a c 1 [cl] and rule2 -c /?v V Q

Current flow (single mode, 2 words)

"john"+"smith" -> -p -c (?a 2 (?a c 1 c -> "JohnSmith" -> -c /?v V Q -> "JoHNSMiTH"

New flow, concat -p -c (?a 2 (?a c 1 [cl].+ M.-c /?v V Q (M always added between them, and in this case also + because we have 1 and/or 2 in the first rule) and then dropping the second -c results in -p -c (?a 2 (?a c 1 [cl] + M /?v V Q

"john"+"smith" -> -p -c (?a 2 (?a c 1 c + M /?v V Q -> "JoHNSMiTH"

A drawback with doing/fixing this is that current --rules-stack run first word through all rules before going on to next word (normal rules do the opposite). This is sometimes wanted.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the --rules-stack path and the preprocessor and rule-engine flow described in the issue. Compare the current two-pass behavior with the proposed concatenated-rule examples, including reject flags, separators, and the different word-order behavior. Done means stacked rules can be processed once while preserving the intended output and handling those edge cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.