asmMBA: Robust Virtualization Obfuscation with Assembly-Based Mixed Boolean-Arithmetic

Commercial virtualization obfuscation tools like VMProtect and Themida, which rely on transforming original code into virtual instructions, have been successfully reverse engineered by attackers. To safeguard the intellectual property of the virtualization obfuscation architecture from reverse engineering, recent works have applied complex Mixed Boolean-Arithmetic (MBA) obfuscation to the handler code responsible for the core functions of the virtualization obfuscation. In this paper, we first show that a state-of-the-art MBA-based protection method such as Loki can be efficiently deobfuscated and then we introduce Loki-Blast. The proposed method effectively simplifies nested MBA expressions, revealing weaknesses in current MBA-based obfuscation methods used in virtualization obfuscation tools. In light of these vulnerabilities, we propose asmMBA, a novel assembly-based MBA obfuscation technique. Applying MBA transformations directly at the assembly level, asmMBA introduces a layer of complexity that complicates the static and dynamic analysis, which enables the software to resist modern deobfuscation tools like MBA-Blast and Chosen-Instruction Attack effectively. Our evaluation shows that asmMBA can generate up to 10⁴² distinct obfuscated versions of a simple program depending on the protection level. This makes it difficult for attackers to acquire reusable knowledge from the target program, and it also significantly increases the complexity of program analysis. We experimentally demonstrate that asmMBA expressions are not deobfuscated by the MBA deobfuscation tool. These results demonstrate that asmMBA provides strong protection against deobfuscation attacks while maintaining manageable performance overhead, making it a practical solution for real-world software protection.