The x86-64 System V calling convention¶
There are limitless options for exactly how to combine registers, memory, and jumps into function calls and returns, but it is absolutely critical that the caller and the callee agree about which options are to be used. Not only will this be different on different architectures, different operating systems will often differ in function calling conventions. Linux on x86-64 uses a very widely adopted calling convention, the System V Application Binary Interface (SysV ABI).
The SysV calling convention is very similar to the 32-bit x86 convention, but there are a few optimizations that make it a little bit more efficient, such as passing some of the parameters in registers rather than memory (faster) and allowing some of the boilerplate to be skipped when it isn’t completely necessary.
Eli Bendersky’s stack frame layout on x86-64 article is a good place to start, and good brief summaries of the calling convention can also be found at the OSDev article on the System V ABI and the Wikipedia article on x86 calling conventions.
One thing to be aware of is caller-saved vs callee-saved registers.
Remember, registers can’t be created at runtime, so when you jump
(or call
) into a function, it uses the same registers for
its work that you were using—your data might get clobbered! As
part of the agreement in the calling convention, some registers are
‘callee-save’, meaning that the callee, the function being called,
promises not to change them—or at any rate to put them back the way
it found them. Other registers are ‘caller-save’, meaning that the
callee can use them how it likes and leave them however is convenient,
and if the caller wants those values saved, it has to do it itself. No
matter which code is responsible for doing it, ‘save’ typically just
means pushing the old value onto the stack and then ‘restoring’ it
later by popping it back into the same register again.