IDA Pro: Hex-Rays decompiler plugin via microcode API

Summary

Build a native C++ Hex-Rays decompiler plugin that simplifies MBA expressions during decompilation using CoBRA's core library. This provides CoBRA's full simplification pipeline (semilinear, MixedRewrite, decomposition, full-width verification) directly in IDA's microcode optimization pass.

Architecture

Model after gooMBA's integration pattern (HexRaysSA/goomba):

lib/ida/
  CobraIDA.cpp          — Plugin entry, hexrays callback, action registration
  MicrocodeDetector.cpp — MBA detection via minsn_t tree walking
  MicrocodeDetector.h
  MicrocodeReconstructor.cpp — Simplified Expr → minsn_t reconstruction
  MicrocodeReconstructor.h

Hook Point

• Register hexrays_callback for hxe_glbopt event (global optimization phase)
• Walk top-level microcode instructions via mba->for_all_topinsns()
• For each instruction, recursively attempt MBA simplification
• Return MERR_LOOP to restart optimization if any simplifications were made

Microcode → Expr Bridge

Map Hex-Rays microcode operations to CoBRA Expr kinds:

Hex-Rays op	CoBRA Expr
m_add	Expr::Add
m_sub	Expr::Add + Expr::Negate
m_mul	Expr::Mul
m_and	Expr::BitwiseAnd
m_or	Expr::BitwiseOr
m_xor	Expr::BitwiseXor
m_bnot	Expr::BitwiseNot
m_neg	Expr::Negate
m_shr	Expr::LogicalShr (constant shift)

Operand types (mop_t): registers → variables, mop_n (numbers) → constants, sub-instructions → recursive descent.

Evaluator

Build evaluator lambda from microcode — assign values to leaf mop_t operands and evaluate the minsn_t tree (similar to goomba's mcode_emu.hpp approach). This enables CoBRA's full-width verification.

Detection Heuristic

Same approach as goomba's is_mba(): count boolean ops (m_and, m_or, m_xor, m_bnot) and arithmetic ops (m_add, m_sub, m_mul). Require both categories present and minimum tree size.

Verification

CoBRA's built-in full-width verification (evaluator + spot checks) replaces goomba's Z3 equivalence proofs. Significantly faster — no Z3 dependency needed for the plugin itself.

Cost Gate

Use ComputeCost/IsBetter to ensure simplified form is actually simpler, same as the LLVM pass.

Build System

• COBRA_BUILD_IDA_PLUGIN CMake option
• Requires IDA SDK path (IDA_SDK_DIR) — not shipped, user must have IDA license
• Links against cobra-core (static)
• Produces cobra_ida.so / cobra_ida.dylib / cobra_ida.dll plugin

Task List

• CMake infrastructure for IDA plugin (SDK detection, plugin target)
• Plugin entry point: hexrays callback, action registration, popup menu
• Microcode → Expr bridge (BuildExprFromMinsn)
• Microcode evaluator (EvaluateMinsn)
• MBA detection heuristic for microcode
• Expr → microcode reconstruction (ReconstructMinsn)
• Cost gate integration
• Variable remapping for aux var elimination
• Testing strategy (IDB-based or standalone microcode tests)

References

• gooMBA — Hex-Rays' official MBA simplifier
• gooMBA hooks hxe_glbopt, walks via for_all_topinsns, uses minsn_visitor_t
• gooMBA returns MERR_LOOP after successful simplification to restart decompiler optimization

Start by reading the proposed lib/ida/ architecture and the gooMBA integration reference, then inspect CoBRA's LLVM pass for ComputeCost/IsBetter and verification behavior. Implementing this issue is complete when the CMake target, Hex-Rays callback, microcode/Expr conversion, evaluator, detection, reconstruction, cost gate, variable remapping, and testing strategy in the task list are covered.