Function Calling Conventions

Terminology

Passing arguments: “passing arguments” is a way of saying that the calling function is writing data in the place where the called function will look for them. Arguments are passed before the call instruction is executed.

Right-to-Left and Left-to-Right: describe the manner that arguments are passed to the subroutine, in terms of the High-level code. For instance, the following C function call:

MyFunction1(a, b);
//will generate the following code if passed Left-to-Right:
push a
push b
call _MyFunction
//and will generate the following code if passed Right-to-Left:
push b
push a
call _MyFunction

Return value: Some functions return a value, and that value must be received reliably by the function’s caller. The called function places its return value in a place where the calling function can get it when execution returns. The called function stores the return value before executing the ret instruction.
Cleaning the stack: when arguments are pushed onto the stack, eventually they must be popped back off again. Whichever function, the caller or the callee, is responsible for cleaning the stack must reset the stack pointer to eliminate the passed arguments.
Calling function (the caller): The “parent” function that calls the subroutine. Execution resumes in the calling function directly after the subroutine call, unless the program terminates inside the subroutine.
Called function (the callee): The “child” function that gets called by the “parent”.
Name Decoration: When C code is translated to assembly code, the compiler will often “decorate” the function name by adding extra information that the linker will use to find and link to the correct functions. For most calling conventions, the decoration is very simple (often only an extra symbol or two to denote the calling convention), but in some extreme cases (notably C++ “thiscall” convention), the names are “mangled” severely.
Entry sequence (the function prologue): a few instructions at the beginning of a function, which prepare the stack and registers for use within the function.
Exit sequence (the function epilogue): a few instructions at the end of a function, which restore the stack and registers to the state expected by the caller, and return to the caller. Some calling conventions clean the stack in the exit sequence.
Call sequence: a few instructions in the middle of a function (the caller) which pass the arguments and call the called function. After the called function has returned, some calling conventions have one more instruction in the call sequence to clean the stack.

Stack Frame

Example Code, sum_till_max.c

#include <stdio.h>
#include <ctype.h>
#include <stdlib.h>

#define MAX (1UL << 20)

typedef unsigned long long u64;
typedef unsigned int u32;

u32 max_add_end = MAX;

u64 sum_till_max(u32 n)
{
  u64 sum;
  n++;
  sum = n;

  if (n < max_add_end)
    sum += sum_till_max(n);

  return sum;
}

int main(int argc, char *argv[])
{
  u64 sum = 0;

  if ((argc == 2) && isdigit(*argv[1]))
    max_add_end = strtoul(argv[1], NULL, 0);

  if (max_add_end > MAX || max_add_end == 0) {
    fprintf(stderr, "Invalid number is specified\n");
    return 1;
  }

  sum = sum_till_max(0);
  printf("sum(0...%u) = %llu\n", max_add_end, sum);
  return 0;
}

Content Assemble Code

(gdb) disassemble main 
Dump of assembler code for function main:
   0x000000000040069b <+0>:	push   %rbp
   0x000000000040069c <+1>:	mov    %rsp,%rbp
   0x000000000040069f <+4>:	sub    $0x20,%rsp
...
   0x0000000000400744 <+169>:	jmp    0x400774 <main+217>
   0x0000000000400746 <+171>:	mov    $0x0,%edi ----------------------------------------------------#1
   0x000000000040074b <+176>:	callq  0x400666 <sum_till_max> --------------------------------------#2
   0x0000000000400750 <+181>:	mov    %rax,-0x8(%rbp)
   0x0000000000400754 <+185>:	mov    0x2008f6(%rip),%eax        # 0x601050 <max_add_end>
...
(gdb) disassemble sum_till_max 
Dump of assembler code for function sum_till_max:
   0x0000000000400666 <+0>: 	push   %rbp ---------------------------------------------------------#3
   0x0000000000400667 <+1>: 	mov    %rsp,%rbp ----------------------------------------------------#4
   0x000000000040066a <+4>: 	sub    $0x20,%rsp ---------------------------------------------------#5
   0x000000000040066e <+8>: 	mov    %edi,-0x14(%rbp)
   0x0000000000400671 <+11>:	addl   $0x1,-0x14(%rbp) ---------------------------------------------#6
   0x0000000000400675 <+15>:	mov    -0x14(%rbp),%eax
   0x0000000000400678 <+18>:	mov    %rax,-0x8(%rbp)
   0x000000000040067c <+22>:	mov    0x2009ce(%rip),%eax        # 0x601050 <max_add_end>
   0x0000000000400682 <+28>:	cmp    %eax,-0x14(%rbp)
   0x0000000000400685 <+31>:	jae    0x400695 <sum_till_max+47>
   0x0000000000400687 <+33>:	mov    -0x14(%rbp),%eax
   0x000000000040068a <+36>:	mov    %eax,%edi
   0x000000000040068c <+38>:	callq  0x400666 <sum_till_max>
   0x0000000000400691 <+43>:	add    %rax,-0x8(%rbp)
   0x0000000000400695 <+47>:	mov    -0x8(%rbp),%rax
   0x0000000000400699 <+51>:	leaveq --------------------------------------------------------------#7
   0x000000000040069a <+52>:	retq ----------------------------------------------------------------#8
End of assembler dump.

Terminology

Stack Frame

Reference