Charles Dardaman

Security engineering, incident response, research.

MalwareTech Shellcode Challenge #1

Today, MalwareTech posted a beginner's reverse engineering challenge on Twitter, and I thought it would be fun to go through it and write up how to do it with IDA and Python.

The ZIP file contained two files, README.txt and shellcode1.exe (which is the challenge file).

README.txt contains the basic rules of the challenge, stating that it's meant to be done statically with IDA and not with a debugger. It also tells us that running the binary will output an MD5 of the flag.

To confirm this, I ran the executable to see what happens. (Please keep in mind that you should never execute code that you don't trust on your system. I ran the file inside of a VM.)

Message box displaying the MD5 hash of the flag

As we can see, it opens up a message box titled “We've been compromised!” and includes the hash of the flag, just as the README said it would.

I opened up IDA Pro to take a better look at the binary. As soon as it loads, the function window shows almost all functions are related to MD5, which makes sense given the size of the binary and our knowledge that it utilizes MD5 to hash the flag.

IDA Pro function list revealing mostly MD5-related functions

Next, I inspected the imports to understand which Windows libraries are being used. The binary can call a message box, allocate space in memory, allocate on the heap, and get the length of a string.

IDA Pro view showing imported functions including MessageBoxA

Returning to the start function, the assembly shows a MessageBoxA call at the bottom. The offset a2bBBb is used in digestString before it is passed to MessageBoxA. This tells me that a2bBBb is the flag I'm looking for. To track it easily, I renamed the offset to flag and continued through the code.

IDA Pro screenshot highlighting the flag string being hashed

Knowing where the flag is located, I moved back to the top of the function and saw that it allocates space on the heap, adds a pointer to the flag into that space, and then adds the length of the flag onto the heap as well.

Assembly allocating heap memory and storing the flag pointer

The next section allocates a 13-byte region of memory and copies whatever is at unk_404068 into that region before calling it as code.

Assembly copying decoder shellcode into executable memory

Jumping to the unk location, IDA initially shows raw bytes. After pressing C to convert it, IDA renders the following readable assembly. This code takes the location of the flag (stored in the heap structure) and rotates each byte left by five bits.

Decoded shellcode performing ROL operations on the flag bytes

This shellcode acts as a decoder, transforming the stored flag before hashing. To replicate the decoding, I tweaked a Python rotate script I found to match the behavior. 

from __future__ import print_function

# max_bits > 0 == width of the value in bits (e.g., int_16 -> 16)

# Rotate left: 0b1001 --> 0b0011
rol = lambda val, r_bits, max_bits: (
    (val << (r_bits % max_bits)) & (2**max_bits - 1)
) | (
    (val & (2**max_bits - 1)) >> (max_bits - (r_bits % max_bits))
)

# Rotate right: 0b1001 --> 0b1100
ror = lambda val, r_bits, max_bits: (
    (val & (2**max_bits - 1)) >> (r_bits % max_bits)
) | (
    (val << (max_bits - (r_bits % max_bits))) & (2**max_bits - 1)
)

max_bits = 8  # For fun, try 2, 17 or other arbitrary (positive!) values

shellcode = ["Insert encoded flag hex here"]

final = ""
for value_hex in shellcode:
    value = int("0x" + value_hex, 16)
    new_value = rol(value, 5, max_bits)
    print("0x%08x << 0x05 --> 0x%08x" % (value, new_value))
    final += chr(new_value)

print(final)

With this script ready, I copied the bytes referenced by the flag location and pasted them into the shellcode list. Running the script reveals the final flag.

← Back to homepage