Getting started with syscalls

What is a system call?

If you look online for MSDN documentation around the word syscall or system call, you might come up empty handed. You might even hit this page thinking you found something of interest. Nope. Not even close. Perhaps some of the best, formal documentation is found in the Windows Internals books when describing system service calls, trapping, and overall system service handling. Windows Internals Chapter 8: System mechanics talks about this, in great detail. Something I might cover way later near the end of this thread.

In general, a system call is an interrupt that will interrupt the system (kernel) and invoke a service routine according to an index value. This value is what is commonly referred to as the system service call number, or system service number, or syscall ID. The kernel will handle the dispatching of the routine.

Perhaps the best source for this is from the Intel Software Developer Manual where it is defined that a syscall is a fast system call to privilege level 0 system procedures.

What generates a system call?

For our purposes here in this thread, syscall is interchangeable with system call. From here on out I will just use the word syscall. These are generated by invoking the instruction syscall or the int 2e instruction. These instructions are heavily found in two low level DLLs: ntdll.dll and win32u.dll.

Where do syscalls come from?

GUI vs Native

Short answer: it depends. Longer answer, a syscall can from from GUI applications that depend on win32u.dll or Native applications where the only dependency is ntdll.dll.

win32u.dll

This module is the lowest level DLL for all GUI applications or rather GUI threads to be the most technically accurate. Something that will become clearer later on is this DLL has a system call table identifier of 0x20. This module implements many syscalls like win32u!NtUserCloseClipboard or win32u!NtUserOpenClipboard. Bottom line here is practically anything that happens from a GUI window will trickle its way down to this module.

On my Dev VM, my version of win32u.dll has 5,411 syscalls implemented.

ntdll.dll

This module is the lowest level DLL for all native functions that are not related to GUIs. Typically you will see a native function with the two-letter prefix Nt, but don’t be fooled, Nt functions are also implemented inside win32u.dll. Bottom line here is anything stemming from native programs, like programs that execute early on in the boot process (the session manager: smss.exe) will trickle its way down to this module.

This graphic shows a nice represenation of the flow a syscall can take depending on the process (GUI vs Native).

How is a syscall structured?

The generic format

All syscalls, no matter if they come from win32u.dll or ntdll.dll, will all have the same kind of structure. You might even think of it as a signature, a signature that can be scanned for in memory just like what is done in a lab for Day 5, or Section 5 for OnDemand students :grin:. Here is what the format looks like:

4c8bd1           mov     r10, rcx
b8c3100000       mov     eax, <some number here>
f604250803fe7f01 test    byte ptr [7FFE0308h], 1
7503             jne     <module_name>!<Some Nt function>+0x15
0f05             syscall 
c3               ret     
cd2e             int     2Eh
c3               ret     
0f1f840000000000 nop     dword ptr [rax+rax]

Let’s break it down a bit more…

mov r10, rcx

For x64 CPUs and of course x64 code, the first parameter is typically found in RCX. You will see that RCX is moved into R10 and this is obviously intentional.

Why is this intentional?

Because the code can eventually execute the syscall instruction, it has to get rid of the RCX value and move it into R10. Further, syscall will destroy the RCX register and load it with the return address. If the MOV is not done, the first parameter will be lost. You can’t use R11 either since it will be clobbered by syscall so the last best option was R10 to hold the first parameter as code transitions into Ring 0.

What’s next?

Let’s continue breaking it down…

mov     eax, <some number here>

<some number here> will be reserved for the number of the syscall to be “called”. Again, this is really an index into a table that we can take a look at later in this series of threads using a kernel debugger.

Here is one full example for a syscall from win32u.dll:

4c8bd1           mov     r10, rcx
b8c3100000       mov     eax, 10C3h

For the function win32u!NtUserOpenClipboard the above number is tied to that function.

I’m going to skip…

test    byte ptr [7FFE0308h], 1
jne     <module_name>!<Some Nt function>+0x15

… for now just because that is getting too deep too soon for where I want this thread.

The next thing that will happen is that syscall will be executed.