Talking to the OS

assembly2-8 established that ring-3 code can't do privileged things directly — it has to ask. This chapter covers how it actually asks, and the answer genuinely splits in two depending on which operating system is listening.

Recap — assembly1-8's Clean Abstraction

LC-3's TRAP was one unified mechanism: a single instruction, one vector table, a small fixed set of OS services (GETC/OUT/PUTS/HALT) defined by the LC-3 specification itself, identical on every LC-3 system that ever exists. Real x86-64 has no equivalent single mechanism — what "asking the OS" even looks like depends entirely on which operating system the code is running under.

Linux — The SYSCALL Instruction

x86-64 Linux provides a dedicated SYSCALL instruction — a fast, purpose-built replacement for the older, slower software-interrupt-based mechanism (INT 0x80) that 32-bit x86 Linux used. The convention: the syscall number goes in RAX, and arguments go in RDI, RSI, RDX, R10, R8, R9.

MOV RAX, 1     ; syscall number for write()
MOV RDI, 1     ; file descriptor 1 = stdout
MOV RSI, msg   ; pointer to the string
MOV RDX, 5     ; length
SYSCALL          ; make the call — return value comes back in RAX

The return value comes back in RAX — the one point of real consistency with assembly2-5's own ordinary function-call convention.

Windows — No Direct Syscalls for Applications

Windows takes a genuinely different approach. It does not expose a stable, documented, directly-invokable syscall interface to ordinary application code the way Linux does — Windows's own internal syscall numbers are considered private implementation detail, and can (and do) change between versions and even updates. Instead, applications call into the Windows API (Win32 API): a large collection of ordinary functions, living in system DLLs like kernel32.dll, which themselves make the real, internal, unstable syscalls on the application's behalf.

The practical consequence: an x86-64 Windows assembly program calling something like WriteFile or ExitProcess does so with an ordinary CALL instruction, using assembly2-5's own Microsoft x64 calling convention, to a function imported from a DLL — not via any direct syscall-style instruction the programmer writes.

A Genuine, Important Divergence

This is the real point: LC-3 had one portable OS abstraction, identical everywhere. x86-64 genuinely has two different models depending on target OS — code written to make Linux syscalls directly has no Windows equivalent to fall back on, and vice versa, even though the underlying CPU instructions covered in every prior chapter of this course work identically on both.

Hands-On Exercises