Go Scheduler

OS Thread ⇐> Core :: G ⇐> M

M is an OS Thread, G is a task that needs to be scheduled to run on M

• M is a thread of execution managed by the OS. Works pretty much like your standard POSIX thread.
  • We can create an unlimited amount of M, e.g. for handling blocking syscalls, but at most P threads can run at any time.
  • M can be w/o P, e.g. if G is making a synchronous syscall.
    • There can be more M than the number of cores.
• G is a goroutine. Includes stack, instruction pointer and other info for scheduling goroutines, e.g. any channel it might be blocked on.
• P is context for scheduling. A localized version of the scheduler which runs Go code on a single thread.
  • each P maintains (1) a local runqueue, to reduce contention, (2) malloc caches, (3) stack caches, (4) other caches.
  • You can think of P as a per-CPU state, i.e. logical processor, which allows us to have a lot of threads (M) with a fixed amount of overhead (runqueues to checked, caches to allocate etc.)
  • Every P is associated with an M.
    • If the task G, which is currently assigned to M1, is going to block, then M1 will be detached from P, and P will be assigned to a new M2.
  • There is exactly GOMAXPROCS P.
    • Implication: the GOMAXPROCS variable limits the number of operating system threads (M) that can execute user-level Go code simultaneously.

M is a physical thread of execution, G is a logical thread of execution. An extra OS Thread is used for NetPoller, which handles async network calls

The struct descriptions of p, m, g is in src/runtime/runtime2.go

Design objective: You never want an M (OS Thread) to go idle, because it will be context switched off the core by the OS scheduler.

M represents OS thread (as it is now). P represents a resource that is required to execute Go code. When M executes Go code, it has an associated P. When M is idle or in syscall, it does not need P. There is exactly GOMAXPROCS P’s. All P’s are organized into an array, that is a requirement of work-stealing. - source: scalable go scheduler

• M and P is both needed to execute a task G.
• M can be w/o P, e.g. if G is making a blocking syscall.
• The default for GOMAXPROCS is runtime.NumCPU()

The ability to turn IO/Blocking work into CPU-bound work at the OS level is where we get a big win in leveraging more CPU capacity over time. This is why you don’t need more OS Threads than you have virtual cores. - source: ardan labs explainer pt 2

• Global runqueue is a FIFO queue used to put preempted goroutines.
  • Goroutines are preempted if it uses the whole timeslice of 10ms. How is it implemented?
• Local runqueue is a FIFO queue with a 1-slot LIFO buffer infront. This LIFO slot is to increase data locality. A scenario that uses this is when a goroutine G1 creates a goroutine G2 and then blocks waiting for the results of G2. In this case, we want to run G2 next, and want to make sure G2 is not being stolen by another processor.

References:

• https://morsmachine.dk/go-scheduler
• https://rakyll.org/scheduler/
  • I think the picture in this article is clearer between the distinction of M and P.
  • Errata:
    • In “Scheduling basics”, 2nd para, 2nd stmt should be: “each P should be assigned to an M. Ms may have no Ps if they are blocked or in a system call.”
• https://morsmachine.dk/netpoller
• https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part2.html
  • The example on Asynchronous syscall (where Gs are swapped out to netpoller) and Synchronous syscall (where M+Gs are swapped out off P) is good.
• 2019 talk from Vyukov, implementer of Go scheduler

---

• When do we know that a G is gonna block? It will block when it does syscalls.
  • The last thing a goroutine do before doing a syscall is wakes up another thread, that will pickup a G from the runqueue.
• How do we know, after G is blocked, when G is ready again.
  • When goroutine finish running syscall, it will be put into the runqueue, and the thread will become idle.