2.1. Microkernels

The VortexOS kernel is a microkernel. Microkernels are notable for their design, providing only minimal abstractions in kernel space. Unlike monolithic kernels, which emphasize kernel-space functionality, microkernels focus more on user-space. The core philosophy behind microkernels is that any component capable of running in user-space should indeed run there. Kernel-space should be reserved for only the most critical functions, such as system calls, process isolation, resource management, inter-process communication (IPC), and thread management.

The main role of the kernel is to facilitate communication and isolation between processes. It should offer minimal abstraction over the hardware—drivers, for instance, should run in user-space when possible. Microkernels offer superior security and are less prone to crashes compared to monolithic kernels. This is due to moving most components into user-space, utilizing separate memory address spaces, reducing the potential for harm to the system. Furthermore, microkernels are highly maintainable, as their smaller codebase results in fewer bugs overall.

As with everything, microkernels come with their own advantages and disadvantages.

Advantages of Microkernels

Improved Modularity and Configuration Monolithic kernels are, as the name implies, monolithic in structure. They lack the fine-grained control that microkernels offer. This is because many essential components are hard-coded directly into the kernel, necessitating modifications when changes are required, such as with device drivers. In contrast, microkernels are inherently modular. You can add, remove, replace, or modify modules during runtime without needing to alter the kernel itself. While modern monolithic kernels have adopted kernel modules to address this issue, they still often require a system reboot.

Enhanced Security Microkernels are considerably more secure than their monolithic counterparts. This stems from the principle of minimality, which aligns with the least privilege concept: each component should have only the permissions absolutely necessary to function. Many security vulnerabilities in monolithic kernels come from services and drivers that run with unrestricted access in kernel mode, exposing the system to various risks. In a monolithic design, drivers can operate without limitations in kernel mode, making the system more vulnerable.

Better System Stability Compared to microkernels, monolithic kernels are more prone to crashes. A faulty driver in a monolithic kernel can bring down the entire system, as both the driver and the kernel share the same memory space. A crash in the kernel cannot be recovered without risking memory corruption, resulting in a system failure (kernel panic). In a microkernel system, drivers are isolated in their own memory spaces, so any crash can be safely contained. In Linux, for instance, drivers frequently encounter issues with dereferencing invalid pointers, often leading to kernel panics. Extensive documentation, such as that found in MINIX, explains how microkernels handle such issues more effectively.

Easier Debugging In microkernels, since most kernel components (e.g., drivers, file systems) run in user-space, bugs in these components don’t result in kernel crashes. This is particularly useful when debugging on physical hardware; if a kernel panic occurs, it is often impossible to save logs that could help identify the bug. In contrast, a bug in a monolithic kernel’s component causes a system-wide crash, locking the system completely. This makes debugging hardware issues significantly more challenging, especially without serial output.

Disadvantages of Microkernels

Slight Performance Overhead Modern operating systems require fundamental security features like memory segmentation and virtualization. Additionally, every process, including the kernel, needs its own stack and register storage. When a context switch occurs, such as during a system call or any other form of inter-process communication (IPC), a series of tasks must be executed:

• Saving the caller’s registers, especially the program counter.

• Reprogramming the memory management unit (MMU) page tables.

• Switching between CPU modes (user mode to kernel mode, and vice versa).

• Restoring the callee’s registers.

While these operations aren’t inherently slower in microkernels, they need to be performed more frequently due to the nature of microkernel design. Since many functionalities are handled by user-space processes, more context switches are required. However, over time, the performance gap between monolithic and microkernels has shrunk considerably, partly due to the smaller codebase in microkernels, which is easier to optimize.

Microkernels vs. Monolithic Kernels

Monolithic kernels offer more built-in abstractions compared to microkernels. The illustration provided in public domain highlights the difference between the two approaches.

Documentation and Further Reading

• Kernel/User-Space Separation Documentation

• Dual Mode Operations in OS

• User Mode and Kernel Mode Switching

• OSDev Technical Wiki

• MINIX Documentation

• MINIX Features and Reliability

• GNU Hurd Documentation

• Fuchsia Documentation

• HelenOS FAQ

• MINIX Papers

• seL4 Whitepapers

• Microkernel Performance Papers

• Tanenbaum-Torvalds Debate

Current State of VortexOS

VortexOS kernel currently contains fewer than 40,000 lines of Rust code. For comparison, MINIX's kernel has about 6,000 lines of C code. However, this comparison does not imply that MINIX is more stable or secure than VortexOS solely based on lines of code. VortexOS includes more advanced features, which naturally results in a larger codebase. In the future, we aim to move even more components to user-space, further minimizing the kernel's size and complexity.

4o