Remember the Registers (or, the ABCs of x86)

I have a friend who is about to learn assembly language on x86. This means understanding how processor registers are used. So, I'm sharing some mnemonics for remembering the registers and their roles on x86. Maybe they will help you, too.

A register, by the way, is just a memory location that is directly (and quickly!) accessible by the processor, usually located in the processor circuitry as opposed to being accessible at a memory address (although this depends on the hardware).

Oh. Okay, so not one of these. Got it.

I'll completely omit floating point (and MMX/SSE) stuff, but I'll briefly mention amd64 for the sake of example.

A, B, C, and D

There are four general purpose registers whose names are basically A, B, C, and D. On old 16-bit Intel processors, they were named AX, BX, CX, and DX. From each of these, you could also access the low eight bits (e.g. AL, as in "A low") and the high eight bits (e.g. AH, as in "A high"), and you still can.

When 32-bit came along, they prefixed them all with an E (meaning "extended"), so now they are EAX, EBX, ECX, and EDX.

And when AMD devised their derivative amd64 architecture, they prefixed them all with an R, presumably meaning "really-friggin'-extended". So, on amd64, they are named RAX, RBX, RCX, and RDX.

The "A" register has a few special roles, but most importantly it is used to hold the return value (i.e. the result) when functions return to their callers.

The "C" register sometimes has a special role, too, that I'll describe next.

So much for A, B, C, and D.

String

Several string instructions (e.g. cmps and movs, stos, lods) have implied operands:

• ESI = Source
• EDI = Destination

• ECX = Counter
• EAX = Value to write (sometimes applicable, sometimes not)

Stack

The stack starts at a high address in memory and as things are added to it, it grows down to lower addresses. The processor keeps track with a stack pointer, and then functions can further use a "base" pointer to point to the boundary between the caller's stack (and two other pieces of data), and their own local variables.

• ESP = Stack Pointer
• EBP = Base Pointer

Instruction Pointer

So special that it's all alone under its own heading:

• EIP = Instruction Pointer

This register points to the instruction that is about to be executed.

Extra Credit: Segment Registers

Remember in Windows 95 when blue screens used to report "CS:EIP = xxxxxxx"? Neither do I, let's pretend I didn't admit that. Anyway, that CS is a segment selector indicating the location of the code segment. Modern operating systems use a flat model, so they're mostly set the same.

• CS = Code Segment
• DS = Data Segment
• ES = "Extra" Segment (meh)
• SS = Stack Segment
• FS & GS = Even more extra segments, using the letters that follow C, D, and E, namely F and G Segments

These registers are 16 bits long and contain integers called segment descriptors; the CPU reads the Global or the Local Descriptor Table (the GDT or the LDT) to find the base address and size of each memory segment indicated by those descriptors. Operating systems commonly use a "flat" model instead of a segmented model, so these may all contain the same value. Since segmentation is largely a non-issue now, the extra segment registers are sometimes used to point to interesting structures. For example: FS => Thread Environment Block (TEB) in userspace Windows 32-bit applications.

Summary

In review:

• EAX, EBX, ECX, EDX = A, B, C, D; Note that the 'A' register holds function return values
• ESI, EDI = Source, Destination (for string operations) - ECX may be the counter and EAX may used, too.
• ESP, EBP = Stack Pointer, Base Pointer
• EIP = Instruction Pointer
• CS, DS, SS, ES, FS, GS = Code, Data, Stack, and Extra segments, followed by F and G Segments

For the authoritative reference on all of this, see the Intel processor manuals.

Happy hackin' :)

The case for reinventing the wheel

I really like to use IDA Pro as my debugger, and shellcode is no exception. Initially I couldn't see why anyone would ever write their own loader for analyzing shellcode. Siko et al released shellcode_launcher.exe along with the Practical Malware Analysis labs, so why rewrite that code? shellcode_launcher.exe does the work of ReadFile / VirtualAlloc / VirtualProtect, et cetera, so I just make that my database and pull in the VirtualAlloc'd memory using IDA Pro's memory snapshot facilities. Then, I go to town.

Well, I changed my tune when I discovered that VirtualAlloc was not receptive to my suggestions for where to allocate memory. (WinDbg: bp <callsite>; g; ed esp <lpAddress>; p). Without a consistent shellcode base address, none of my annotations from the IDA memory snapshot I took were lining up with the actual shellcode in subsequent debug sessions.

Edit January 14th, 2018: At this point, we have a choose-your-own-adventure on our hands:

• If you use remote debugging, and/or you like to see IDA Pro annotations superimposed over your debugger session, and your shellcode itself allocates additional memory and executes code there, then you might be better off reading my fireeye.com blog article titled Debugging Complex Malware that Executes Code on the Heap.
• If you don't use remote debugging, then you might be satisfied capturing snapshots of your debugging VM at critical points in the debug session so you can iteratively debug and understand the shellcode.
• Finally, if your shellcode does not execute additional code on the heap and you just want to give it a uniform memory map in which to iteratively debug it, then read on...

For simple cases, you can reinvent the wheel and write your own shellcode loader to force shellcode to live at the same virtual address each time you debug it. But no need to start from scratch; here's the path of least resistance...

Assuming you have the shellcode as a raw binary, use xxd's feature of outputting shellcode as a C include file:

xxd -i myshellcode > myshellcode.c

That gives you a hexdump in C form:

unsigned char myshellcode[] = {
    0x55, 0x8b, 0xec, 0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01,
    0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, 0x23, 0x45, 0x67,
    ...
    0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef, 0x01, 0x23, 0x45, 0x67,
};
unsigned int myshellcode_len = 4242;

So now, write your loader:

#include "myshellcode.c"

typedef void (*fptr)(void);

int
main(void)
{
    fptr sc = (fptr) myshellcode;
    __asm int 3 ; Safety - so I don't execute this on my analysis box (or worse!)
    sc();
}

Then do this (in an SDK prompt):

cl.exe loader.c

If you don't have Visual Studio, just get Microsoft's compiler for Python 2.7.

After compiling and linking, you'll get this:

I was worried about execute permissions when calling into my shellcode, but happily, it Just Works, perhaps because I ran cl.exe directly without using Visual Studio to specify its usual flags. The program loads in IDA as a PE-COFF, it can be debugged using IDA's debugging plugins, and the shellcode is always at the same address (in my case, unk_40A000). Therefore, you can annotate the shellcode without using the IDA Pro memory snapshot facilities to save it from a debugger session, and (this is the important part) without worrying that VirtualAlloc will return a different address during the next debug session, rendering your annotations less useful to you. The same applies to breakpoints: they will actually work, from session to session. That makes life easier.

CPUID, SMSW, and Other Delights

I wrote a quick and dirty utility to collect info a la redpill, nopill (props to Danny Quist but I can't find that whitepaper anymore!), etc. Nothing really novel about it, but I thought others may find it useful for researchy scenarios. I used it to investigate a hypervisor running on an Intel microprocessor, so each output line includes an indication of whether VMX appears to be supported. My intent was to train a Bayes learner to identify systems that are lying about whether they support VMX (thus likely detecting a hypervisor), similar to a previous project of mine, except in the course of this project, that became so very unnecessary.

Here is a snippet of its output:

So hex. Much flashy.

This tool works by creating and affinitizing a thread to each logical CPU in the system, executing a few compiler intrinsics and assembly functions, and outputting the desired information for each CPU. Like most research code I post, this tool is only as complete as I needed it to be for my own purposes. Therefore, it does not support 32-bit platforms, does not collect the value of the SLDT instruction from each processor, and is not meant for AMD microprocessors. If you can tolerate all that, then the source code is here:

https://github.com/strictlymike/cpuinfo

For more information about CPUID and using microprocessor instructions, I thought the book Professional Assembly Language Programming was very helpful, and of course, the Intel 64 and IA-32 microprocessor manuals are the authoritative reference on all things Intel x86 and x64.

Flare-On 2015 #2 Write-Up, Part 2

Well, I guess I can post the script and results, now that Flare-On has been over for over a month! I'll continue from where I left off in my previous article.

Previously, I implemented a function to skip the ReadFile call and stuff arbitrary passwords into the challenge binary's input stream. By experimenting with this, I discovered a potential tell that might allow me to brute force the password by counting the number of times a particular instruction was executed. To make use of this, I needed to execute the binary several times with different passwords (aaaa..., abaa..., acaa...), moving on to the next character each time an additional sahf instruction was executed.

To do this, I knew I would first need to figure out which characters are valid for email addresses. I settled on a variant of the information available from a stackoverflow inquiry:

g_valid = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.#-_~!$&\'()*+,;=:%{|}\\/?@'

The next step was to initialize a buffer that would serve as my brute force password. How long could the password be? Based on the length comparison at the start of the password processing function...

It looked as if the program was seeking a 37-byte password. So, I whipped one up:

g_pw = list('a' * 37)

In order to move through candidate passwords, I iterated through valid characters for each character position of the brute force password:

    for pos in range(0,37):
        for c in g_valid:
            g_pw[pos] = c

To measure my progress, I executed the binary with one breakpoint set to stuff the brute force password into the program's input buffer and the other breakpoint set to count sahf instructions:

            trace = vtrace.getTrace()
            trace.execute(sys.argv[1])
            trace.addBreakpoint(MyBreakpoint(0x400000 + 0x1046, skipread))
            trace.addBreakpoint(MyBreakpoint(0x400000 + 0x10ae, count_iters))
            trace.setMode("RunForever", True)
            trace.run()

And finally, I compared the number of sahf breakpoints hit in each successive occurrence against the previous maximum. If an additional breakpoint was hit, I advanced to the next character position in the brute force password and began working on that character. The script in its entirety looked something like this:

 1 # coding: utf-8
 2 import vtrace
 3 import vdb
 4 import sys
 5 import struct
 6 
 7 # Offsets for d88dafdaefe27e7083ef16d241187d31 *very_success.exe:
 8 # cs:0x4010ae <-- sahf instruction: single iteration of password character scan
 9 #   baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa - breakpoint hits once
10 #   aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa - breakpoint hits twice
11 # Trying brute force by counting breakpoint hits
12 # cs:0x401046 <-- call kernel32!ReadFile, len at ebp-4
13 # cs:0x40104c <-- return from kernel32!ReadFile, len at ebp-4
14 # ds:0x402159 <-- gPasswordFromCONIN$[]
15 
16 # Props to: http://www.limited-entropy.com/stuff/drmless.py.txt
17 class MyBreakpoint(vtrace.Breakpoint):
18     def __init__(self, address, callback):
19         vtrace.Breakpoint.__init__(self, address)
20         self.address = address
21         self._cb = callback
22 
23     def notify(self, event, trace):
24         self._cb(trace)
25 
26 # Count of times sahf instruction was encountered in character scan loop
27 g_bp_count = 0
28 
29 # Count how many times sahf instruction was encountered in character scan loop
30 def count_iters(trace):
31     global g_bp_count
32     g_bp_count = g_bp_count + 1
33 
34 # Skip past ReadFile() call and tell the program that it received the bytes in
35 # the array g_pw
36 def skipread(trace):
37     trace.setProgramCounter(0x400000 + 0x104c)
38     print "\nTrying " + ''.join(g_pw)
39     tmp = ''.join(g_pw)
40     trace.writeMemory(0x400000 + 0x2159, tmp)
41     trace.writeMemory(trace.parseExpression("rbp-4 & 0xffffffff"), struct.pack("@i", 37))
42 
43 if len(sys.argv) != 2:
44     print "Usage: test.py filename"
45     sys.exit(1)
46 
47 # http://stackoverflow.com/questions/2049502/what-characters-are-allowed-in-email-address
48 g_valid = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.#-_~!$&\'()*+,;=:%{|}\\/?@'
49 
50 g_pw = list('a' * 37)
51 g_pw.append("\0")
52 g_breaks = 0
53 while True:
54     for pos in range(0,37):
55         for c in g_valid:
56             g_pw[pos] = c
57             g_bp_count = 0
58 
59             trace = vtrace.getTrace()
60             trace.execute(sys.argv[1])
61             trace.addBreakpoint(MyBreakpoint(0x400000 + 0x1046, skipread))
62             trace.addBreakpoint(MyBreakpoint(0x400000 + 0x10ae, count_iters))
63             trace.setMode("RunForever", True)
64             trace.run()
65             print "Breaks " + str(g_breaks)
66 
67             if (g_bp_count > g_breaks):
68                 g_breaks = g_bp_count
69                 break

The result was a little bit like a cinematic depiction of a password being brute forced, because the characters kept rolling by and locking in as the attack progressed:

Very Success!

In the screenshot above, you can see the output of the CTF binary (very_success.exe) interspersed with the script indicating what password was being tried (e.g. "Trying a_Little_b1t_baaaa...").

Using vtrace made this an incredibly cool learning experience, but it wasn't without its pitfalls. For instance, I was really excited when I found this snippet in the help for the Trace class:

     call(self, address, args, convention=None)
         Setup the "stack" and call the target address with the following
         arguments.  If the argument is a string or a buffer, copy that into
         memory and hand in the argument.

However, when I tried using this, I was disappointed to learn that I was out of luck:

Well, crap.

It could be that running a 32-bit binary in a WoW64 environment causes this condition, but I didn't have time to investigate, so I just went with executing the program over and over and over again, which worked fine.

There was also a flaw in my own process. I noticed that at various character positions, my assumptions broke down and the brute forcer ran off into the weeds. I suspect this was a function of the binary itself and not of the instrumentation tool (vtrace). With a little guesswork and some script adjustment, I was able to resume from various points in the process and pull the remainder of the password out of the binary:

a_Little_b1t_harder_plez@flare-on.com

After I submitted the flag, FLARE sent me the next binary and I celebrated by abandoning the CTF altogether, as I ultimately knew might happen due to my busy life. But I was glad I took the approach of figuring out how to create a scalable and repeatable instrumentation process for analyzing and abusing binaries. There are other tools out there, but vtrace is unique in that it doesn't require you to compile a DLL, and it allows you to make use of Python to instrument applications. Very useful!

Flare-On 2015 #2 Write-Up, Part 1

I've been away from CTFs for a while, but at the urging of some colleagues, I took a brief look at this year's Flare-On challenge. I won't post a complete write-up today because I wouldn't want to give away the flag itself before the Flare-On challenge ends on September 8th. But maybe this article will give you ideas if you are still working on any of the challenges.

Because I knew I wouldn't be able to stay with it, I focused on using my time to learn how to do new things. The first challenge was easy and educational because FLARE produced a relevant write-up in 2014 that includes an IDA script to decode XOR-encoded data, a technique which has also been documented in Practical Malware Analysis. The second challenge, however, featured some unusual instructions in its decode-and-check loop:

The sahf, adc, and popf mnemonics were unfamiliar to me, however this did not look like a case of disassembly desynchronization. I started by reading the Intel manuals...

...but my short attention span got the better of me. So, I began using WinDbg to see why the sahf and adc instructions were used, what effect they would have on the program's control flow, and whether I was wrong about the disassembly desync. After a single iteration of this, I decided that my laziness and short attention span would require me to create a repeatable substrate for an iterative process of observation and experimentation: I needed automate the instrumentation.

There are several options for this. The painstaking way would be to pick up the code I wrote based on Microsoft's debugger documentation and a CodeProject article from 2011. Other alternatives included experimenting with DynamoRIO or Intel's PIN (which I will get into another time). And then there's Visi's vtrace. This latter tool is Python-based, so it was pretty appealing. I decided to give it a shot.

First, I downloaded the vtrace/vdb distribution (I might have grabbed it from here). Then I took a look at Visi's article to get started. I also found an interesting presentation and a PlaidCTF write-up that provided me with some additional ideas.

Reversing in IDA gave me some offsets to start with:

# Offsets for d88dafdaefe27e7083ef16d241187d31 *very_success.exe:
# cs:0x4010ae <-- sahf instruction: single iteration of password character scan
# cs:0x401046 <-- call kernel32!ReadFile, len at ebp-4
# cs:0x40104c <-- return from kernel32!ReadFile, len at ebp-4
# ds:0x402159 <-- gPasswordFromCONIN$[]

With this, I wrote a vtrace script that used the MyBreakpoint class in the presentation above to manipulate program execution and mark how many times the sahf instruction was hit. Specifically, I wrote a function to skip the ReadFile() Win32 API call and instead feed a password of my choosing to the program:

34 def skipread(trace):
35     print "Broken at eip = " + hex(trace.getProgramCounter()) + " (skipread)"
36     trace.setProgramCounter(0x400000 + 0x104c)
37     print "Trying " + ''.join(pw)
38     tmp = ''.join(pw)
39     # buf = pw.encode('utf-8')
40     trace.writeMemory(0x400000 + 0x2159, tmp)
41     print "ebp: " + hex(trace.parseExpression("rbp"))
42     print "ebp-4: " + hex(trace.parseExpression("rbp-4"))
43     print "[ebp-4]: " + hex(trace.parseExpression("poi(rbp-4) & 0xffffffff"))
44     trace.writeMemory(trace.parseExpression("rbp-4 & 0xffffffff"), struct.pack("@i", 37))

I executed this with a few different passwords. Of particular interest were the results with all "a" characters and all "b" characters:

I noticed something useful here: the breakpoint I had set on the sahf instruction to mark the number of loop iterations was hit once more when processing the "a" password than when processing the "b" password. Looking back on my notes, I see that I was enthusiastic about this:

Holy **** yes, saw password processing breakpoint hit twice for 'a'x37 and only once for 'b'*37

The reason for my excitement was that this probably meant the password could be brute forced. So, that is exactly what I did. Sure, maybe I skipped the whole part where you analyze and understand the code. But there's more to life than intimately understanding every Intel instruction I've ever encountered. In my next article, I'll post the scripts and talk about the results.