InfiniBand

InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices.

Kernel
Users of Mellanox hardware MSX6710, MSX8720, MSB7700, MSN2700, MSX1410, MSN2410, MSB7800, MSN2740, and MSN2100 need at least kernel 4.9.

The following kernel options must be activated:

IP over InfiniBand (IPoIB)
Users of InfiniBand can also use it to carry IP-networking packets, thus allowing it to replace ethernet in some cases. From at least kernel version 4.10 onwards users can compile IP-over-IB in-kernel ( CONFIG_INFINIBAND_IPOIB ). However if that's not the case and the support is compiled as kernel module it may be possible that the module isn't automatically loaded. The module is named ib_ipoib:

InfiniBand network interfaces are usually named as :

Performance tuning
When using IP over InfiniBand ther performance is usually low by default. This is because the default mtu of each InfiniBand IP interface is set to a low value.

Automatic
The most conveivent way is to chance mtu automatically when kernel adds the interface to the system. Before mtu can be changed the mode of the interface must be changed to 'connected' from 'datagram' Next example uses udev rules to accoplish that.

It has been reported that the rule above does not work. Users with this problem may use the following instead:

Manual
Mode and mtu can also be changed manually in run time. Next commands assume the interface in question is named ib0.