NUMA: Cores, memory, and the distance between them

Edera's recent updates to Xen-based virtualization address a longstanding performance issue related to Non-Uniform Memory Access (NUMA). Specifically, the company developed a stack of changes that make the virtualization NUMA-aware, from within the guest operating system to the paravirtual I/O drivers, dom0, and back to the hypervisor's view of host hardware. This change was likely prompted by a scenario where two identical virtual machines on the same host, running the same workload, exhibited a consistent 20% performance difference due to the placement of memory on one side of an interconnect from the CPUs. The affected virtual machine's operator had no in-guest controls to rectify the issue.

The development of NUMA-aware virtualization by Edera is significant in the context of modern server architecture. NUMA emerged as a solution to the limitations of Uniform Memory Access (UMA) in multi-socket servers. As the industry moved beyond single-socket commodity servers, manufacturers like AMD and Intel introduced NUMA to allow each socket to have its own memory controller. For instance, AMD's Opteron, launched in 2003, and Intel's Nehalem, launched in 2008, adopted NUMA. Today, AMD's EPYC processors and Intel's Xeon parts with Sub-NUMA Clustering (SNC) feature multiple NUMA nodes per socket, complicating the traditional "one socket, one node" mental model.

The implications of Edera's NUMA-aware virtualization are substantial for cloud and virtualization providers. By optimizing virtual machine performance in NUMA environments, Edera can help reduce performance variability and contention for interconnect bandwidth. This is particularly crucial as modern servers often present multiple NUMA nodes, and the cost of remote DRAM access can be 1.5x to 3x that of local access. As the industry continues to evolve, with increasing core counts and complex interconnects, solutions like Edera's will be essential for maintaining performance and efficiency in virtualized environments.