Low-level VM Protection

Parent Previous Next

Low-level VM Protection.

A small VM may be inserted into the executable as a replacement for a section of native assembler instructions. This could for example mean replacing a given function with a VM by using a tool which creates a modified compiled binary file.


Low-level VMs can constitute a significant degree of complexity for a cracker - when properly implemented - making it very difficult to analyze and modify the protected functions.


However, certain pitfalls exist that may compromise the overall security. When starting at the lowest point (the native instructions) and moving to a higher level (the VM), there is a risk that it doesn't raise the complexity for the cracker as much as one might expect.


It is extremely important the VM execution constitutes a big leap away from the original assembler instructions, making it impossible for the cracker to analyze and interpret the functionality. If this leap is not big enough and the cracker begins to understand what is happening during execution, the security could be compromised simply by leaving out the VM or injecting other instructions into the executable.


It is also important to know exactly, how certain functions or sections of code are replaced and what happens when jumping in and out of the protected areas, e.g. when using calls to library functions outside the protected areas. The use of standard functions can reveal a lot. Ideally1 you should be using in-lining on multiple functions and turn them into a big block of protected code and eliminate standard calls and text messages. Having a fragmented structure - with some functions being protected and others not - can feed the cracker with enough information to perform a successful attack.


Note 1: Often good advice in general and for low-level VM protection in particular.


Assessment.

Using a low-level VM to protect a function means that at no point does the protected code exist in memory in its original pristine form. This makes analysis very hard and modifications to a protected function almost impossible.


However, it does not protect against bypassing the protected functions or injecting instructions to modify certain corners of the program in general. And the overall security very much depend on the implemented VM methodology of hiding the original instructions plus the possible weaknesses from a fragmented internal structure with jumps in and out of protected code. Furthermore it does not encrypt the memory involved.


Cracking can be done using ordinary tools e.g. debugger, disassembler, memory monitor, hosted emulator etc. but it will most likely be very difficult and time consuming for every application being attacked.


Due to these considerations, an application containing functions protected by low-level VMs will probably possess a medium level of security and a rather low risk of failing badly due to the expected uniqueness of the implemented VMs.