Dynamic 'compiler' work (main work task)

All subtasks will be linked to this main Issue. ANY new user issues started should also be linked into this topic. Current known subtasks and bug reports: #2800 - #3389 - (#3125 this one may not be part of this task conglomeration) - #1746
As discussed with @magnumripper off-list, here is the plan

• 1 Build the scripts as best as possible (like is done today, just make some improvements where they can be made).

• 2 at time of 'linkage' to dyna, the compiler will run a couple samples.

• 3 if the samples 'pass' and crack, then the linkage will happen like it does today (to the native dynamic_fmt format).

• 4 if the samples fail, then I will instead link to a 'new' format.

This 'new' format will perform oSSL using a recursive decent parser against the already built recursive decent parser tree. This will be very similar to the code "in" the compiler which generates the test strings. This code will not have the limitations of the 2 input and 2 output buffers which dynamic formats are limited to. This is the problem when we try to actually 'emit' the code from the compiler. Going from a recursive decent tree, into an iterative language, which has only 2+2 variables, and NO stack, will only work 'so far'. Thus once there is a format built that properly uses the RDP tree of data, any valid expression (no matter how complex) will work fine. It will be slow however, single iteration ssl, little to NO chance for optimizations. I do think I will emit a warning message at the start of the run, listing to the user that this format is non optimal (and direct them to john ML or documentation), and listing that writing a proper dynamic format would likely run much faster, if there is much work at all to do with hashes of this expression

• 5 I will also work to have more expressions work within the compiler. Many 'could' work fine, but no matter how much better I make it, it is VERY easy to make a valid expression which CAN NOT work within the limitations of dynamic. Any format which would require 3 temp 'registers' can not be done with the current dynamic. So something like this

md5($s.md5($p.md5($p.$s)).md5($s.$p))

simply can not be done with the existing flat dynamic format. It 'can' be done within dynamic, using all of the input buffers AND the output buffers. But that usage does not work in flat mode (without changes), which would cause dynamic code base to grow substantially. Also, even using all 4 variables (both inputs and both outputs), it is still trivial to generate expressions which require 5 temp registers to complete. An expression that complex (or more complex), CAN NOT be handled by the dynamic format AT ALL. So no matter what, there will be expressions which dynamic of today CAN NOT handle. So the new RDP dynamic format, will have to be done, else the 'compiler' will always have bug reports like we see that show up on a regular basis.