GNU Debugger (also known as GDB) is a great utility when trying to understand program.

set disassembly-flavor intel

start

run

r

x/10i $rip
x/gx $rsp

display/10i $rip

print $rsp
set $rsp = 0

break *

We can automate this process too by making gdb script

start
set disassembly-flavor intel
r
disas main

Pwntools is a python utility used for testing exploits.

Other than some built-in functions, it works same as python. So having a python understanding is PLUS here.

from pwn import *

Every program is a process. If its on your host machines you simply write its name and if it is a remote process you simply write remote and give ip:port combination in the parameters.

p = process('/path/to/program')
p = remote('127.0.0.1', 8080)

Once we get the process, we read or write some data.

So if we want to read it first.

p.recv()
p.recvuntil(b'some string')

And then we put/write exploit

p.send(b'data')
p.sendafter(b'some string', b'aaaa')

Computers store data in bytes.

And 1 Byte is equal to 8 bits.

So 5 in decimal is equal to 00000101 in binary. If the sign bit¹ is 0 we call it a positive number and if the sign bit is set (or 1) we call it a negative number.

But the problem (for ALU) with the above approach is 0 will have two distinct values. +0 will be equal to 00000000. And -0 would be equal to 10000000.

That’s why we have two’s complement.

Suppose we subtract 2 from 1

0000 0001 = 1
0000 0010 = 2

# This operation is not possible, so we borrow a bit

1 0000 0001 = now 1 becomes (256 + 1)
  0000 0010 = 2

0 1111 1111 = 255

# If the number is unsigned it's 255, if it's signed it's -1 (as 1 - 2 = -1)

So the trick here is if you want signed value of any unsigned number just subtract it with 256 (if it’s a byte).

Suppose if we want a two’s complement (signed number) of 255. It’s 255 - 256 = -1

And the opposite is true. If we want unsigned value of any signed number we add it with 256.

Let’s say we want unsigned number of -1. It’s 256 + (-1) = 255

A few other things to note: a single byte is equal to 2 nibbles (4 bits each). Total values unsigned 1 byte can hold is 2^8 (0 to 2^8 -1). And for a signed byte it’s from - 2^7 to +2^7 -1

Also rather than doing these calculations on a calculator we can use the following python3 methods to make things a little bit more efficient.

>>> hex(0b11101110)
'0xee'
>>> bin(0xee)
'0b11101110'
>>> int(0b11101110)
238
>>> bytes.fromhex('ee')
b'\xee'
>>> b'\xee'.hex()
'ee'
>>> int('ee', 16)
238