IOMMU SWIOTLB

This article provides Article description::instructions for enabling and configuring Input Output Memory Management Unit and the Software Input Output Translation Lookaside Buffer for use with the Linux kernel.

Today's computing uses a method of partitioning memory and each device such as a graphics card, PCI device, or USB device has to have memory mapped to be accessed by the device or application.

Traditionally IOMMU was used for memory mapping. This is set up when the system is initialized and can not be dynamically changed as the system is running so chip manufacturers such as Intel and AMD developed more advanced memory management methods.

In the Linux kernel we can manipulate the IOMMU using new mechanisms provided by SWIOTLB for Intel and others for architectures from AMD. 64-bit systems have enabled a huge amount of memory to be used in by the system and this memory needs mapping before it can be used. These kinds of terms are used across the Enterprise area of computing, particularly the Virtual-Machine sector but they can be used by anyone running a Linux kernel.

IOMMU
This is Input Output Memory Management Unit. In every system this hardware is integrated into a north bridge controller which sets up the memory and is programmed by the firmware on your main-board. In recent years manufacturers have stopped integrating this as a North-Bridge chip and integrated it into the CPU itself. This is why if you want to upgrade your memory speed, type and so on you are now required to not only change the motherboard but the CPU as well.

Regardless the kernel needs to set up and read the mappings to be able to use your system memory efficiently.

Kernel
Move to the kernel directory and open the menuconfig utility:

Set the following options:

The above will allow the kernel to control the mappings in the Memory Mapping controller.

Build and install the kernel:

Configuration
The following options for controlling aspects of the memory mapping will need to be added to the kernel command-line in order to take effect. Doing so varies depending on the system's bootloader. Common bootloaders include GRUB2 and Lilo. More information on bootloaders can be found here.

Intel systems
Intel generally adopts "an-always-enable-it-if-it-is-supported" rule so most options are to turn off or disable the IOMMU functions.

SWIOTLB
Software Input Output Translation Lookaside Buffer is an Intel technology which sort of bypasses the IOMMU and allows for a much more configurable memory management interface. Without going into the deep complexity of how this works, page tables are cached in the Lookaside Buffer reducing the need to constantly access physical RAM to map memory. This technology is also referred to as a bounce buffer as the physical address of the memory map is held in this virtual space of and IO is bounced between the physical IO and the Physical memory by this virtual lookaside buffer.

This allows the memory mapping to be carried out quickly and have a physical memory space available for use much faster than if it had to be created physically in RAM and presented to the system as usable.

Each IO TLB is referred to as a slab, this can be found in the kernel header source file:

So this means 1MB would be 8 slabs and the value used as the boot parameter is in slabs NOT size.

SWIOTLB for high input output (such as graphics)
For decades the problem has existed in that how would you get data in and out of the CPU and RAM quickly and efficiently especially for high throughput devices like file IO and graphic cards, etc.

Unfortunately the system is not only having to deal with that IO but many tasks all at the same time, the CPU and RAM may be very fast but if it cannot get the data out by either network, USB, storage device, or onto a screen via a graphics card it is a waste of time having a fast multiprocessing system.

Normally the system holds 4MB for normal operation and allows the rest to be used by other devices. The problem is that if a device overlaps or overflows into another then the system panics and can't deal with it. Many new devices like Nvidia graphics cards and SCSI controllers have drivers now that allow the IOMMU values they use to be set.

There is no safe way this value can be set (adjusted) automatically because of the diversity of hardware configurations possible on the market. This means the end user has to design and build the system and decide for each use case the best setting for the system.

If one set a large SWIOTLB then one would need to instruct the driver of a device to utilize the larger amount of memory mapping buffer. Some hardware physically control this in the BIOS while others do not provide any control. For the most part, newer high end hardware permit system administrators to control this by modifying the above kernel options.

Some drivers try to automatically control this but as mentioned above can cause stability issues even kernel panic.

Simply setting a large SWIOTLB will not mean a faster IO will be achieved; the hardware must be instructed to use it. Rule of thumb is if 64MB is available then set a maximum remap IO for the driver of 4MB less. This would be 60MB in this case. If 128MB then max remap for the driver would be 124MB and so on.