Syscalls, Mach Traps & MIG

System calls are the only gateway from userspace into the kernel. Each syscall handler is a potential attack surface.

Entry Points into the Kernel

ARM64 Syscall Mechanism

User: system call
  → SVC #0x80 instruction
    → ARM64 exception to EL1
      → Exception vector (VBAR_EL1 + offset)
        → Trap handler (osfmk/arm64/trap.c)
          ├── x16 >= 0 → BSD syscall (bsd/kern/syscalls.master)
          ├── x16 < 0  → Mach trap (osfmk/kern/syscall_sw.c)
          └── x16 == -31 → mach_msg_overwrite_trap (fast path)

BSD Syscalls

// Entry: SVC #0x80, x16 = syscall number (positive)
// Arguments: x0-x5 (up to 6 args)
// Return: x0 = return value, carry flag = error

// Example: read(2)
//   x16 = 3 (SYS_read)
//   x0 = fd
//   x1 = buf (userspace pointer)
//   x2 = count

Syscalls as attack surface:

• setsockopt / getsockopt – complex option parsing, historically buggy
• ioctl – device-specific commands
• proc_info – process information queries
• necp_* – network extension control
• posix_spawn – process creation
• mac_* – MAC framework operations
• shm_open / sem_open – shared memory / semaphores (used for kread/kwrite primitives)

Mach Traps

// Entry: SVC #0x80, x16 = negative trap number
// Defined in: osfmk/kern/syscall_sw.c

// Key Mach traps:
mach_msg_overwrite_trap    // -31: IPC message send/receive
_kernelrpc_mach_vm_allocate_trap  // -10: VM allocation
_kernelrpc_mach_vm_deallocate_trap // -12: VM deallocation
_kernelrpc_mach_port_allocate_trap // -16: Port allocation
_kernelrpc_mach_port_insert_right_trap // -19: Insert port right
task_self_trap             // -28: Get own task port
thread_self_trap           // -27: Get own thread port
host_self_trap             // -29: Get host port

MIG (Mach Interface Generator)

MIG generates RPC stubs for Mach IPC interfaces. The server side runs in the kernel.

.defs files → migcom → server stubs (kernel) + client stubs (user)

Key MIG subsystems:
  Subsystem 200: mach_host   (host operations)
  Subsystem 400: mach_port   (port operations)
  Subsystem 3400: task        (task operations)
  Subsystem 3600: thread_act  (thread operations)
  Subsystem 2800: vm_map      (VM operations)
  Subsystem 2900: UNDRequest  (user notification)

MIG attack surface:

• MIG handlers parse complex messages with port rights and OOL data
• Type confusion is possible when the MIG-expected type differs from the actual type
• Size validation bugs in struct parsing
• Port right handling errors leading to UAF, double-free

Resources

• XNU source: bsd/kern/syscalls.master, osfmk/kern/syscall_sw.c
• XNU source: osfmk/mach/*.defs (MIG definitions)