HLIL handling of VMProtect

Version and Platform (required):

• Binary Ninja Version: 3.1.3469
• OS: Windows
• OS Version: 11
• CPU Architecture x64

Bug Description:
I'm reversing the Adylkuzz sample and the VMProtect handlers decompile some weird ways. There are a few issues:

Identifying relocated pointers

VMProtect uses a mov eax, 0x00000000 instruction and this 0x00000000 value is referenced in the app's relocations, so the VM can build offsets off of this both to jump to handlers and also to convert any current pointers back to RVAs. This gets optimised out in the HLIL (because X + 0 is equivalent to X), but this hides some key information. IDA handles this by highlighting the immediate value 0x00000000 in red in the disassembly, for example

Junk code removal
The VMEnter function pushes all the registers and flags, decrypts VIP and initialises the stack, virtual registers, and stream cipher. It keeps track of registers that have been pushed and trashes them afterwards, and also trashes them between uses. Take this snippet for example:

005becbc  50                 push    eax {var_10}
005becbd  c6c4cf             mov     ah, 0xcf  // junk
005becc0  8bdb               mov     ebx, ebx  // junk
005becc2  51                 push    ecx {var_14}
005becc3  0fbfc0             movsx   eax, ax  // junk
005becc6  57                 push    edi {var_18}
005becc7  55                 push    ebp {var_1c}
005becc8  66bb287f           mov     bx, 0x7f28  // junk
005beccc  0fbfda             movsx   ebx, dx  // junk
005beccf  9c                 pushfd   {var_20}
005becd0  // relocated to where binary is loaded, we can leave this as 0
005becd0  b800000000         mov     eax, 0x0

We see eax doesn't get used for anything between being pushed at 005becbc and being overwritten at 005becd0, yet we still get the assign at 005becbd in the HLIL:

005becbd      eax:1.b = 0xcf  // junk

Overriding register use

There are other registers that are modified and binary ninja seems to rely on x86 register conventions for this, so this isn't strictly incorrect but it would be nice to be able to hint which registers we care about as input as output (is this something we can do in a plugin adding our own calling convention?)

For example, I can set esp as an argument but it still resolves mov [esp+eax], ecx to *(&__return_addr + eax) = *arg2 instead of arg5[eax] = *arg2. Also I know that esi, esp, ebp and ebx will be persisted across functions, so is there a good way to make sure the HLIL knows to preserve this (but also ignore others)?

Handling of indirect jumps

The function ends with this

005bed5f  03f8               add     edi, eax
005bed61  57                 push    edi {var_e8}
005bed62  c3                 retn    

And the HLIL decompiles it as this

005bed54      int32_t eax_8 = rol.d(not.d((*(ror.d(esi_4 ^ 0x27e9128c, 1) - 3) ^ (ror.d(esi_4 ^ 0x27e9128c, 1) + 1)) - 0x5ca20a41) - 0x40d54c06, 2)
005bed61      int32_t var_e8 = 0x5bed29 + eax_8
005bed62      return eax_8

It recognises the assignment of var_e8, but it greys it out as dead code rather than identifying that this is the new return address (so is effectively a jump instruction)

Intermediate variables

This could be an existing feature that I've not found, but it would be good if we could choose when to roll assignments onto a single line (when there's only one output) and when we'd like to use an intermediate variable (e.g. when there are multiple outputs or when we're trying to extract an intermediate value). For example, the final equation in the VMEnter function resolves like this:

005bece8      int32_t esi_4 = neg.d(arg5 + 1)
...
005bed54      int32_t eax_8 = rol.d(not.d((*(ror.d(esi_4 ^ 0x27e9128c, 1) - 3) ^ (ror.d(esi_4 ^ 0x27e9128c, 1) + 1)) - 0x5ca20a41) - 0x40d54c06, 2)

This great! But, from my research I know that the value of esi after 005bed01 is most interesting to me and I'd like to store that as an intermediate variable, and also, extracting this ror.d(esi_4 ^ 0x27e9128c, 1) - 3 simplifies the equation on line 005bed54 of the HLIL.

I have the opposite problem in the next VM handler:

VMHandler:

005babbd  81ee01000000       sub     esi, 0x1
005babc3  0fb606             movzx   eax, byte [esi]
005babc6  f9                 stc       // junk
005babc6  // junk
005babc7  // junk
005babc7  22cf               and     cl, bh
005babc9  6685ed             test    bp, bp  // junk
005babcc  32c3               xor     al, bl
005babce  6681c91538         or      cx, 0x3815  // junk
005babd3  f7c6a74be74d       test    esi, 0x4de74ba7  // junk
005babd9  660fbaf956         btc     cx, 0x56  // junk
005babde  f6d0               not     al
005babe0  2ca3               sub     al, 0xa3
005babe2  d0c0               rol     al, 0x1
005babe4  0fbfce             movsx   ecx, si  // junk
005babe7  0fb7c8             movzx   ecx, ax  // junk
005babea  fec0               inc     al
005babec  d2c9               ror     cl, cl  // junk
005babee  d0c8               ror     al, 0x1
005babf0  32d8               xor     bl, al  {0xa7}
005babf2  8b4c2500           mov     ecx, dword [ebp]
005babf6  f5                 cmc       // junk
005babf6  // junk
005babf7  8dad04000000       lea     ebp, [ebp+0x4]
005babfd  890c04             mov     dword [esp+eax], ecx {__return_addr}
005bac00  0fb7c1             movzx   eax, cx
005bac03  0facc858           shrd    eax, ecx, 0x58
005bac07  81ee04000000       sub     esi, 0x4
005bac0d  98                 cwde    
005bac0e  8b06               mov     eax, dword [esi]
005bac10  6681fde315         cmp     bp, 0x15e3
005bac15  33c3               xor     eax, ebx
005bac17  84d1               test    cl, dl
005bac19  6681fd3d1d         cmp     bp, 0x1d3d
005bac1e  8d80bff55da3       lea     eax, [eax-0x5ca20a41]
005bac24  85ca               test    edx, ecx
005bac26  f8                 clc     
005bac27  f7d0               not     eax
005bac29  f5                 cmc     
005bac2a  f6c609             test    dh, 0x9
005bac2d  2d064cd540         sub     eax, 0x40d54c06
⋯005bac32  c1c002             rol     eax, 0x2
005bac35  f5                 cmc       // junk
005bac35  // junk
005bac36  33d8               xor     ebx, eax
005bac38  663bf4             cmp     si, sp {__return_addr}  // junk
005bac3b  03f8               add     edi, eax
005bac3d  ffe7               jmp     edi

This produces the following HLIL:

005babc6      // junk
005babc7      // junk
005babc9      // junk
005babc7      arg1.b = arg1.b & nullptr
005babc9      // junk
005babcc      uint32_t eax
005babcc      eax.b = *(arg3 - 1) ^ 0xa7
005babd9      // junk
005babde      eax.b = not.b(eax.b)
005babe0      eax.b = eax.b - 0xa3
005babe4      // junk
005babe2      eax.b = rol.b(eax.b, 1)
005babe7      // junk
005babec      // junk
005babea      eax.b = eax.b + 1
005babec      // junk
005babee      eax.b = ror.b(eax.b, 1)
005babf0      int32_t ebx
005babf0      ebx.b = 0xa7 ^ eax.b
005babf6      // junk
005babfd      *(&__return_addr + eax) = *arg2
005bac35      // junk
005bac38      // junk
005bac3d      jump(arg4 + rol.d(not.d((*(arg3 - 5) ^ ebx) - 0x5ca20a41) - 0x40d54c06, 2))

The lines from 005babcc to 005babfd could be rolled into a single HLIL AST, and at a minimum should have different SSA variables assigned. Is this perhaps due to the use of AL confusing things?

VMEnter Disassembly

005becad  56                 push    esi {var_4}
005becae  f7d6               not     esi  // junk
005becb0  0fce               bswap   esi  // junk
005becb2  6687f6             xchg    si, si  // junk
005becb5  52                 push    edx {var_8}
005becb6  53                 push    ebx {var_c}  {0xea6bdba7}
005becb7  0f9cc3             setl    bl  // junk
005becba  8bf4               mov     esi, esp  // junk
005becbc  50                 push    eax {var_10}
005becbd  c6c4cf             mov     ah, 0xcf  // junk
005becc0  8bdb               mov     ebx, ebx  // junk
005becc2  51                 push    ecx {var_14}
005becc3  0fbfc0             movsx   eax, ax  // junk
005becc6  57                 push    edi {var_18}
005becc7  55                 push    ebp {var_1c}
005becc8  66bb287f           mov     bx, 0x7f28  // junk
005beccc  0fbfda             movsx   ebx, dx  // junk
005beccf  9c                 pushfd   {var_20}
005becd0  // relocated to where binary is loaded, we can leave this as 0
005becd0  b800000000         mov     eax, 0x0
005becd5  66bf8d5e           mov     di, 0x5e8d  // junk
005becd9  50                 push    eax {var_24}  {0x0}
005becda  660f46dd           cmovbe  bx, bp  // junk
005becde  8b742428           mov     esi, dword [esp+0x28 {arg5}]
005bece2  46                 inc     esi
005bece3  660fb3d5           btr     bp, dx  // junk
005bece7  f8                 clc       // junk
005bece7  // junk
005bece8  f7de               neg     esi
005becea  660fbdde           bsr     bx, si  // junk
005becee  81f68c12e927       xor     esi, 0x27e9128c
005becf4  f9                 stc       // junk
005becf4  // junk
005becf5  // junk
005becf5  0fbaf59d           btr     ebp, 0x9d
⋯005becf9  d1ce               ror     esi, 0x1
005becfb  660fb6df           movzx   bx, bh  // junk
005becff  8bdd               mov     ebx, ebp  // junk
005bed01  46                 inc     esi
005bed02  66c1d72a           rcl     di, 0x2a  // junk
005bed06  8d3406             lea     esi, [esi+eax]
005bed09  c1f5a1             shl     ebp, 0xa1  // junk
005bed0c  8bec               mov     ebp, esp {var_24}
005bed0e  660d3c25           or      ax, 0x253c  {0x0}  // junk
005bed12  8da42440ffffff     lea     esp, [esp-0xc0]
005bed19  8bde               mov     ebx, esi
005bed1b  b800000000         mov     eax, 0x0
005bed20  1bfd               sbb     edi, ebp {var_24}  // junk
005bed22  6681c77735         add     di, 0x3577  // junk
005bed27  2bd8               sub     ebx, eax
005bed29  8d3d29ed5b00       lea     edi, [0x5bed29]
005bed2f  6633c3             xor     ax, bx  // junk
005bed32  8db6fcffffff       lea     esi, [esi-0x4]
005bed38  66c1c872           ror     ax, 0x72  // junk
005bed3c  d2f4               shl     ah, cl  // junk
005bed3e  8b06               mov     eax, dword [esi]
005bed40  33c3               xor     eax, ebx
005bed42  f8                 clc       // junk
005bed42  // junk
005bed43  8d80bff55da3       lea     eax, [eax-0x5ca20a41]
005bed49  f7d0               not     eax
005bed4b  84db               test    bl, bl  // junk
005bed4d  f8                 clc       // junk
005bed4d  // junk
005bed4e  2d064cd540         sub     eax, 0x40d54c06
005bed53  f5                 cmc       // junk
005bed53  // junk
005bed54  c1c002             rol     eax, 0x2
005bed57  f8                 clc       // junk
005bed57  // junk
005bed58  // junk (flags ignored)
005bed58  6681ff575c         cmp     di, 0x5c57
005bed5d  33d8               xor     ebx, eax
005bed5f  03f8               add     edi, eax
005bed61  57                 push    edi {var_e8}
005bed62  c3                 retn    

VMEnter HLIL

005becae      // junk
005becad      int32_t var_4 = arg4
005becae      // junk
005becb0      // junk
005becb2      // junk
005becb0      _bswap(not.d(arg4))
005becb2      // junk
005becb5      int32_t var_8 = arg2
005becb6      void* const var_c = 0xea6bdba7
005becb7      int32_t ebx  // junk
005becb7      bool s
005becb7      bool o
005becb7      ebx.b = s != o
005becba      // junk
005becbc      int32_t eax
005becbc      int32_t var_10 = eax
005becbd      eax:1.b = 0xcf  // junk
005becc0      // junk
005becc3      // junk
005becc2      int32_t var_14 = arg1
005becc6      int32_t edi
005becc6      int32_t var_18 = edi
005becc7      int32_t var_1c = arg3
005becc8      int32_t ebx_1  // junk
005becc8      ebx_1.w = 0x7f28
005beccc      // junk
005beccf      bool c
005beccf      bool p
005beccf      bool a
005beccf      bool z
005beccf      bool d
005beccf      int32_t var_20 = (o ? 1 : 0) << 0xb | (d ? 1 : 0) << 0xa | (s ? 1 : 0) << 7 | (z ? 1 : 0) << 6 | (a ? 1 : 0) << 4 | (p ? 1 : 0) << 2 | (c ? 1 : 0) << 0
005becd5      edi.w = 0x5e8d  // junk
005becda      // junk
005becd9      int32_t var_24 = 0
005bece3      arg3.w = arg3.w & not.w(1 << modu.w(arg2.w, 0x10))  // junk
005bece7      // junk
005bece8      int32_t esi_4 = neg.d(arg5 + 1)
005becea      int32_t eflags  // junk
005becea      uint16_t temp0
005becea      temp0, eflags = _bit_scan_reverse(esi_4.w)
005becf4      // junk
005becf5      // junk
005becfb      // junk
⋯005becf9      bool c_1 = unimplemented  {ror esi, 0x1}
005becff      // junk
005bed02      // junk
005bed02      edi.w = rlc.w(edi.w, 0x2a, c_1)
005bed09      // junk
005bed0e      // junk
005bed0e      int32_t eax_1
005bed0e      eax_1.w = 0x253c
005bed2f      // junk
005bed38      // junk
005bed3c      // junk
005bed3c      int16_t eax_2
005bed3c      eax_2:1.b = ror.w(0 ^ (ror.d(esi_4 ^ 0x27e9128c, 1) + 1).w, 0x72):1.b << arg1.b
005bed4b      // junk
005bed4d      // junk
005bed53      // junk
005bed57      // junk
005bed58      // junk (flags ignored)
005bed54      int32_t eax_8 = rol.d(not.d((*(ror.d(esi_4 ^ 0x27e9128c, 1) - 3) ^ (ror.d(esi_4 ^ 0x27e9128c, 1) + 1)) - 0x5ca20a41) - 0x40d54c06, 2)
005bed61      int32_t var_e8 = 0x5bed29 + eax_8
005bed62      return eax_8

Lots of questions here, thanks so much for your time.