《加密与解密》is a book written by a couple of master at kanxue.com that tells you about techniques in the software reverse engineering. I bought this book a few years ago, but I couldn’t understand the content at the time. After a few years on the shelf, I sold it two years ago. But recently I read this book again because I wanted to decrypt a piece of software.
OK. I only want to talk about Chapter 20, 《虚拟机的设计》. It’s “how to design a virtual machine” in english. To be precise, it teaches you how to design a virtual machine software, like the well known VMProtect, to protect your software. This chapter is not easy to understand because the author omitted many details. So, I’d like to write some notes down.
Part 1 Generate a clean exe
OK, at first, let’s generate an exe by Visual Studio 2022 based on the code below.
1 | int main() { |
Then, open this exe with IDA Pro(32bit).
We find that the generated assembly code is complex although our program is the simplest!
We need a clean exe for subsequent research, but how to do that? I read an article called “How to write your own packer”. In this article, the author listed some options for generating a clean exe, which have all changed in Visual Studio 2022.
Well, in Visual Studio 2022, we just need to tweak a few options.
First, fill Properties->Linker->Advanced->Entry Point with “main” so that the compiler does not generate redundant code in the entry point.
Second, set “Randomized Base Address” to “No”, and “Fixed Base Address” to “Yes”, which helps us to dynamically debug the program in OllyDbg.
OK, now, let’s generate the exe again and open it with IDA Pro, we’ll see that the resulting assembly code is very clear!
There are also a couple of options that may help you.
The fist option is Properties->Linker->Optimization->References. Setting this to “No” will ensure that your function is preserved in the final exe.
For example, if we generate an exe base on the following code
1 | void crack_me() { |
When opened with IDA Pro, the function crack_me will not be found because it is not used by any of the code in the main function, so the compiler omits it when generating assembly code. Changing “References” to “No” will force the linker to preserve the function crack_me.
The second option is Properties->C/C++->Code Generation->Security Check.
The third option is “Enable Enhanced Instruction Set”, which is also under Code Generation. Turning this option off will force the compiler to generate simple code instead of code using advanced but more efficient assembly instructions such as xmm. This option is useful because you may be not sure if your disassembly engine understands these advanced instructions, so I encourage you turn this option off.
Part 2 Technical details in the code
The author gives a VMP he wrote for a software reverse engineering competition, although the VMP was written over a decade ago. The disassembly engine used by the VMP is from OllyDbg. So, /J should be added as an additional option into Properties->C/C++->Command Line.
The idea of the VMP is to translate assembly instructions of a function that need to be protected (such as crack_me in the code above) into virtual instructions and running those instructions in the virtual machine.
The function of the virtual machine is to read virtual instructions one by one in memory and dispatch them to different handlers. So, the virtual machine is actually a set of assembly instructions that can be executed by CPU and the virtual instructions are actually a set of bytes, not assembly instructions.
So, the VMP consists of two parts. The first part is the virtual machine. The second part is a program which is able to translate the assembly instructions into virtual instructions and adding the virtual machine and virtual instructions into the final exe.
In the VMP, the VCommand contains the assembly instructions of the virtual machine, which is the first part mentioned above, and the InterpretHandler contains the code that can convert assembly instructions to virtual instructions.
So what does the VMP actually do?
Let’s take a look at the first instruction at the entry point
1 | push ebp |
Due to the instruction will be protected and executed by the virtual machine, so the result of the instruction should be that the vebp is stored on the virtual stack.
It will be translted into following virtual instructions
1 | DPushReg32 vebp_idx |
There is a context which contains registers like EAX, EBX, and etc, but it is used and maintained by the virtual machine and addressed by EDI. For ease of distinction, we use veax and vebx and etc to refer to these registers. Similarly, there is also a stack used and maintained by the virtual machine and addressed by EBP. Note that the virtual stack is only be used in those situations where the real stack could be used, such as calling a function.
And, ESI points to the next byte in the stream of virtual instructions.
So, in the code above, vebp_idx is the index of the vesp in the VM context, and DPushReg32 pushes the value of the vebp onto the REAL stack, not the virtual stack! So, after executing the first instruction, the real stack will look like this
1 | vebp |
And ESP points to it.
So, what does the vPush do?
1 | .bug:00404632 vPush: ; DATA XREF: .bug:00404030↑o |
The first instruction indicates that vebp is moved to eax.
The second instruction indicates that EBP is shifted up so that deposit a parameter onto the virtual stack.
The third instruction indicates that the vebp is deposited onto the virtual stack.
Let’s ignore the fourth instruction for now. I’ll discuss it later.
The final instruction indicates to some checks are to be performed to prevent the real stack from being so close to the VM context.
Well, after executing the first two instructions, vebp is stored in the virtual stack. That’s what we want!
Then the virtual machine will pop the vebp on the real stack into the VM context.
The DPushXXX and DPopXXX pairs are used to ensure that the virtual registers are not be changed by the virtual instructions.
What does the vSaveEsp do?
After executing DPushReg32 and DPopReg32, the real stack is balanced, but what about the virtual stack?
We know that the EBP has been subtracted by 4. The vSaveEsp simply sets vesp to EBP to record the new stack top!
1 | .bug:00404627 vSaveEsp: ; DATA XREF: .bug:0040402C↑o |
OK. now let’s consider the second instruction
1 | mov ebp,esp |
It will be translted into following virtual instructions
1 | DPushReg32 vesp_idx |
After the first two instructions, the real stack looks like
1 | vebp |
what does the vMov do?
1 | .bug:00404649 vMov: ; DATA XREF: .bug:00404034↑o |
Looking at the code above, it is interesting to note that we can regard eax and ebx as references to parameters. For example, eax is a reference to vebp, and ecx is a reference to vesp, so
the fifth instruction ‘mov eax, ecx’ means to set vebp to vesp. Why? Because at the end of this code, the values of eax and ecx are stored back on the real stack, and then the value, which is the new vebp,in the real stack is stored back in the VM context due to the next virtual instruction “DPopReg32 vebp_idx”.
DFree simply adds 4 to the ESP to keep the real stack balanced, since the second value in the real stack is useless.
OK. now let’s consider another instruction
1 | sub esp,8 |
It will be translted into following virtual instructions
1 | DPushReg32 vesp_idx |
After the first two instructions, the real stack looks like
1 | 8 |
what does the vSub do?
1 | .bug:00404672 vSub: ; DATA XREF: .bug:00404038↑o |
Well, it means that vesp is subtracted by 4. But vesp represents the top of the virtual stack, and now EBP is clearly not equal to vesp, so use vRestoreEsp to set EBP to vesp.
OK. That’s all the confusion I encountered while reading the source code. I’m sure you can write clearer code after understanding how the VMP works. I’d like to do this, but I don’t have the time at the moment. By the way, the author has made some modifications to OllyDbg’s disassembly engine to get the extra information needed by the VMP. I discovered that udis86 can do this as well. Anyway, even though the code for VMP is not very good, the author should be thanked.