Demystifying DEP (Data Execution Prevention)

We all (cybersecurity space) heard about DEP, some might don’t know and want to understand how it works and what its purpose. This post is specially for new people or non-technical people that want to understand DEP.

Why DEP was created ? #

DEP stands for Data Execution Prevention and was created in a response to exploits that used to execute code on the stack or heap. DEP marks these memory regions as non-executable which causes the exploits to fail.

ROP (Return Oriented Programming) Technique #

In a response to DEP, ROP technique was created allowing to bypass DEP protection by re-using executable code that is already present in the application’s memory, by combining the pieces of re-usable code is possible to executable arbritary code in memory by calling Win32 Apis and executing shellcode, the pieces of code is often called by security researchers as ROP Gadgets.

Understanding x86 #

We need to understand each major registers in x86 architecture, the CPU utilizes a set of registers for performing operations. registers are small, fast storage locations within the CPI and its used to hold data temporarily during computation.

General-Purpose Register #

It’s important to understand the purpose of each register because when working on building the ROP Gagets, choosing the ROP Gagets combination will become easier.

EAX (Accumulator Register) #

Primarily used for arithmetic and data manipulation operations. It can also be used for I/O port access. Specific Uses: Often used to hold the results of arithmetic operations like addition, subtraction, and multiplication.

EBX (Base Register) #

Used as a base pointer for memory access, especially in older addressing modes. Specific Uses: Holds the base address of data in memory. Can be used as a general-purpose register in modern contexts.

ECX (Counter Register) #

Primarily used as a loop counter. Specific Uses: Used in loop and string operations, where it typically holds the count for loop iterations or the length of a string.

EDX (Data Register) #

Used for I/O operations and multiplication/division operations. Specific Uses: Often used to hold the high-order word in multiplication and division operations, as well as for I/O port addresses.

ESI (Source Index) #

Used as a pointer for source data in string operations. Specific Uses: Points to the source in memory for operations like movsb, movsw, movsd (string move instructions).

EDI (Destination Index) #

Used as a pointer for destination data in string operations. Specific Uses: Points to the destination in memory for operations like movsb, movsw, movsd.

EBP (Base Pointer) #

Purpose: Used to point to the base of the stack frame. Specific Uses: Helps in stack frame referencing for local variables and function parameters.

ESP (Stack Pointer) #

Points to the top of the current stack. Specific Uses: Keeps track of the call stack, managing function call and return addresses.

Applying what we know about ROP (Return Oriented Programming) #

As mentioned above ROP is re-usable executable code to perform arbitrary execution code in memory, how that works ? below is an example of ROP Gagets.

The small piece of ROP Gagets below retrieves the memory address for kernel32!VirtualProtect function, VirtualProtect.

What all of them have in common ? they all end with a “ret” assembly instruction, in assembly the ret instruction will execute the code instruction on top of the stack. To build a ROP Gaget the instruction must end with a “ret” after performing the operation the “ret” will execute the next instruction just a like chain of instructions.

The first instruction 0x100fcba2 (pop esi ; ret ;) which will store the address hex value: 0x100fcba2 in esi and execute the next rop gaget instruction at 0x100fa8be. The next rop gaget instruction 0x100fa8be (pop ecx ; ret;), we are now storing the 0x10406154 in ecx and the “ret” instruction will execute the next rop gadget instruction 0x100f864c (and ecx, esi ; pop esi ; mov eax, ecx ; pop ebx ; retn 0x0004), this operation will peform an and operation between value stored in ecx and esi, the result is the memory address of VirtualProtect imported by vulnserver.exe executable.

The next rop gaget instruction 0x10187f21 (mov ecx, [ecx] ; mov eax, ecx ; ret ; ) will deference the address stored in ecx and store the value in ecx. We successfully accomplished our goal to retrieve through rop gaget the memory address for kernel32!VirtualProtect function.

Here you can check the full DEP bypass POC I developed for vulnserver.exe

Note.: The junk for (reg32) is necessary for each pop instruction if the memory address contains several pop instrunctions without having a push instruction for each pop, you need to address that by accounting with junk values otherwise your next valid gaget will be popped into these register instead of being executed by the ret instruction.

# ROP CHAIN
# Patching VirtualProtect
# 0x100fcba2: pop esi ; ret ; (1 found)
# 0xf406154:  VirtualProtect for and operation
# 0x100fa8be: pop ecx ; ret ; 
# 0x10406154: VirtualProtect for and operation
# 0x100f864c: and ecx, esi ; pop esi ; mov eax, ecx ; pop ebx ; retn 0x0004 
# 0x42424242: Junk for esi
# 0x42424242: junk for ebx
# 0x10187f21: mov ecx,  [ecx] ; mov eax, ecx ; ret ; 
# 0x42424242: junk for retn 0x0004

Humble Recommendation #

When developing ROP gagets something I learned through experience and understanding well the purpose of each register is that a lot can be accomplished by looking for eax, ecx, esi and esp registers, the ebp,edx,ebx,edi are important as well, but if you look for all available instructions the 4 registers I mentioned you should be able to do a lot and accomplish your goal. Most applications and modules will use these registers excessively allowing us to perform different operation/combinations and facilitating us to achieve our goal.

Again, if you have not checked yet, check my DEP bypass POC Here