Unpacking Malware: From UPX to Custom Crypters

A systematic approach to identifying and unpacking packed malware - covering UPX, Themida, custom packers, and manual unpacking techniques with x64dbg.

Why Malware Gets Packed

Packing serves two purposes: it compresses the binary (reducing file size) and it obscures the original code from static analysis. The original PE is compressed/encrypted and a small unpacking stub is prepended. At runtime, the stub decompresses the original code into memory and transfers execution to the original entry point (OEP).

Identifying Packed Binaries

Entropy Analysis

Packed sections have high entropy (close to 8.0 for encrypted, 6.5-7.5 for compressed). Normal code is 5.0-6.5.

# Check with DIE (Detect It Easy) or pestudio
python3 -c "
import math, sys
data = open(sys.argv[1], 'rb').read()
freq = [0]*256
for b in data: freq[b] += 1
entropy = -sum((f/len(data)) * math.log2(f/len(data)) for f in freq if f)
print(f'Entropy: {entropy:.2f}')
" sample.exe

Section Names

Common packer signatures:

Section Name	Packer
UPX0, UPX1	UPX
.themida	Themida/WinLicense
.vmp0, .vmp1	VMProtect
.aspack	ASPack
.nsp0	NSPack
.enigma1	Enigma Protector

Import Table

Packed binaries have minimal imports - often just LoadLibrary, GetProcAddress, and VirtualAlloc. The real imports are resolved at runtime by the unpacking stub.

DIE (Detect It Easy)

diec sample.exe
# Output: UPX(3.96)[NRV2B,brute] → identified packer and version

Unpacking UPX

UPX is the simplest - it has a built-in unpack option:

upx -d packed.exe -o unpacked.exe

But sometimes UPX headers are corrupted intentionally to prevent upx -d. In that case, manually unpack:

Load in x64dbg
Set breakpoint on VirtualAlloc (the stub allocates memory for unpacked code)
After allocation, set hardware breakpoint on the allocated region
Run until the breakpoint fires - the OEP jump is nearby
Follow the jump to OEP, dump with Scylla

Manual Unpacking: The Universal Method

This works for any packer. The goal: find the moment after unpacking when execution transfers to the original code.

Step 1: Find the OEP

Tail jump method - Packers end with a JMP to the OEP. Look for:

JMP to a far address (outside the packing stub)
PUSH addr + RET combination (push OEP, ret to it)
CALL followed by stack manipulation

ESP trick (x86) - At the entry point, set a hardware breakpoint on [ESP]. The packer saves and restores registers. When it restores ESP to its original value, the breakpoint fires and you’re at the OEP.

Memory breakpoint method - Set a memory-on-execute breakpoint on the .text section. The packer unpacks into .text, and when execution reaches it, you’re at the OEP.

Step 2: Dump the Process

Once at the OEP, use Scylla (integrated into x64dbg):

Scylla → OEP field → current EIP
IAT Autosearch → find import table boundaries
Get Imports → resolve all imports
Dump → save the unpacked PE
Fix Dump → patch the dump with correct imports

Step 3: Fix the PE

The dumped PE needs corrections:

Section alignment and sizes
Entry point update
Import directory fix (done by Scylla)
Remove overlay data (if any)

Dealing with Anti-Debug in Packers

Sophisticated packers actively resist debugging.

Common Checks

// IsDebuggerPresent
if (IsDebuggerPresent()) ExitProcess(0);

// NtQueryInformationProcess - DebugPort
DWORD debugPort = 0;
NtQueryInformationProcess(GetCurrentProcess(), 7, &debugPort, sizeof(debugPort), NULL);
if (debugPort) ExitProcess(0);

// Timing check
DWORD t1 = GetTickCount();
// ... code ...
DWORD t2 = GetTickCount();
if (t2 - t1 > 100) ExitProcess(0);  // Debugger slowdown detected

Bypasses in x64dbg

ScyllaHide plugin - Automatically patches all common anti-debug checks
TitanHide - Kernel-level anti-anti-debug
Manual NOP - Patch the conditional jump after each check to always fall through

Multi-Stage Packers

Advanced malware uses multiple layers:

Stage 1: Outer packer (custom XOR stub)
  → Decrypts Stage 2 in memory
Stage 2: Inner packer (Themida/VMProtect)  
  → Unpacks Stage 3 in new allocation
Stage 3: Shellcode loader
  → Reflective DLL injection of Stage 4
Stage 4: Final payload (RAT, stealer, etc.)

Each stage requires separate unpacking. Set breakpoints on VirtualAlloc/VirtualProtect to catch each transition.

Automated Unpacking

For known packers, automated tools save time:

unipacker - Generic unpacker using emulation (Unicorn engine)
PE-sieve - Detects and dumps in-memory implants
Mal-Unpack - Dynamic unpacker using API monitoring
ANY.RUN / Triage - Cloud sandboxes that dump unpacked payloads

Indicator Extraction Post-Unpack

Once unpacked, extract IoCs:

Strings: floss (FLARE’s advanced string extraction)
Network indicators: IPs, domains, URLs, user-agents
File paths and registry keys
Mutex names (often unique per campaign)
Crypto keys and configurations

The ability to unpack any binary is the most fundamental malware analysis skill. Master the ESP trick, memory breakpoints, and Scylla - they work against 90% of packers you’ll encounter.