Calling conventions and the stack

In order for subroutines to be general functions (in particular, to support recursion), it is necessary to be able to cheaply allocate storage at runtime for calling a function and giving it the necessary parameters and return address, and space for its local variables. Dynamic memory allocation is tricky in general, but luckily function calls have structure we can exploit: a function may call other functions, and so on, but it will not return until all of the calls it made have returned. Thus, cleaning up after functions follows the pattern ‘last in, first out’ (LIFO), and we can do all of our allocations on the data structure known as a stack.

There is a 32-bit-architecture-centered introduction to how the stack is structured and used for function calling written by Eli Bendersky. I have also used a great 32-bit guide by Paul Krzyzanowski, but be aware that it uses the AT&T assembly syntax rather than the Intel syntax for its example code. (In 32-bit mode, you’ll see e.g. ebp instead of rbp and some other small differences, but the 32-bit mode function calling convention is a little simpler, enough to be worth starting there.)

The main takeaway you should get from these articles is that the boilerplate at the top and bottom of a function should no longer be mysterious to you. For example, if you compile the shortest C program (int main(){}), you get this; familiar?

main:
    push    rbp
    mov     rbp, rsp
    mov     eax, 0
    pop     rbp
    ret

Now you should be able to take your time line by line, and figure out what each of them is doing and why. And, in disassembled code in the future, when you see memory locations being accessed relative to rbp (or rsp) you will know you are seeing parameters and local variables, and be able to reverse engineer what the functions are doing.

You have attempted of activities on this page