Diskless nodes

This HOWTO will help you create setup diskless nodes with Gentoo Linux.

About this HOWTO
This HOWTO will help you setup diskless workstations based on the Gentoo Linux distribution. We intend to make this as user friendly as possible and cater to the Linux newbie, because every one of us was one at a certain point :) While an experienced user could easily tie the multiple HOWTOs available on diskless nodes and networking together we hope that this guide can ease the installation for all interested users, geeks or not.

What is a diskless machine?
A diskless machine is a PC without any of the usual boot devices such as hard disks, floppy drives or CD-ROMs. The diskless node boots off the network and needs a server that will provide it with storage space as a local hard disk would. From now on we call the server the master, while the diskless machine gets called the slave (what's in a name :). The slave node needs a network adapter that supports PXE booting or Etherboot; check Etherboot.org for support listings. Most modern cards support PXE and many built-in adapters on motherboards will also work.

Before you start
You should have Gentoo installed on your master node and enough space on the master to store the file systems of the slave nodes you want to host. Also make sure you have one interface to the internet separated from the local area connection.

About kernels
The kernel is the software that sits between your hardware and all other software you have loaded on your machine, essentially the heart of a kernel based operating system. When your computer is started, the BIOS executes the instructions found at the reserved boot space of your hard drive. These instructions are typically a boot loader that loads your kernel. After your kernel has been loaded all processes are handled by the kernel.

For more information on kernels and kernel configuration you might want to check out the kernel HOWTO.

Configuring the master kernel
The master kernel can be as large and as customized as you would like but there are a few required kernel options you need to select. Go into your kernel configuration menu by typing:

You should get a grey and blue GUI that offers a safe alternative to manually editing the file. If your kernel is currently functioning well you might want to save the current configuration file by exiting the GUI and type:

Go into the following sub-menus and make sure the listed items are checked as built-in (and NOT as modular). The options show below are taken from the 2.6.10 kernel version. If you use a different version, the text or sequence might differ. Just make sure you select at least those shown below.

If you want to access the internet through your master node and/or have a secure firewall make sure to add support for iptables:

If you want to use packet filtering, you can add the rest as modules later. Make sure to read the Gentoo Security Handbook Chapter about Firewalls on how to set this up properly.

After you have re-configured the master's kernel you will want to rebuild it:

Then add an entry for that new kernel into or  depending on which bootloader you are using and make the new kernel the default one. Now that the new bzImage has been copied into your boot directory all you will have to do is reboot the system in order to load these new options.

About the slave kernel
It is recommended that you compile the slave kernel without any modules, since loading and setting them up via remote boot is a difficult and unnecessary process. Additionally, the slave kernel should be as small and compact as possible in order to efficiently boot from the network. We are going to compile the slave's kernel in the same place where the master was configured.

To avoid confusion and wasting time it is probably a good idea to backup the master's configuration file by typing:

Now we will want to configure the slave's kernel in the same fashion we configured the master's kernel. If you want to start with a fresh configuration file you can always recover the default file by typing:

Now go into the configuration GUI by typing:

You will want to make sure you select the following options as built-in and NOT as kernel modules:

Now the slave's kernel needs to be compiled. You have to be careful here because you don't want to mess up the modules (if any) you have built for the master:

Now create the directory on the master that will be used to hold slaves' files and required system files. We use but you may choose any location you like. Now copy the slave's bzImage into the directory:

Configuring a preliminary slave file system
The master and slave filesystems can be tweaked and changed a lot. Right now we are only interested in getting a preliminary filesystem of appropriate configuration files and mount points. First we need to create a directory within for the first slave. Each slave needs it's own root file system because sharing certain system files will cause permission problems and hard crashes. You can call these directories anything you want but I suggest using the slaves IP addresses as they are unique and not confusing. The static IP of our first slave will be, for instance,  :

Various configuration files in need to be altered to work on the slave. Copy the master's directory onto your new slave root by typing:

Still this filesystem isn't ready because it needs various mount points and directories. To create them, type:

Most of these "stubs" should be recognizable to you; stubs like,  or  will be populated when the slave starts, the others will be mounted later. You should also change the file to reflect the hostname of the slave. Binaries, libraries and other files will be populated later in this HOWTO right before you attempt to boot the slave.

Even though is populated by   later on, you need to create the  entry. If not, you will receive the error "unable to open initial console".

About the DHCP server
DHCP stands for Dynamic Host Configuration Protocol. The DHCP server is the first computer the slaves will communicate with when they PXE boot. The primary purpose of the DHCP server is to assign IP addresses. The DHCP server can assign IP addresses based on hosts ethernet MAC addresses. Once the slave has an IP address, the DHCP server will tell the slave where to get its initial file system and kernel.

Before you get started
There are several things you will want to make sure are working before you begin. First check your network connectivity:

You will want to make sure you have have an eth0 device running. It should look something like this:

A properly working eth0 device

It's important that it says MULTICAST, if it doesn't then you will have to recompile your kernel to include multicast support.

Installing the DHCP server
If your network does not already have a DHCP server installed you will need to install one:

If your network already has a DHCP server installed you will have to edit the configuration file to get the PXE boot to function correctly.

Configuring the DHCP server
There is only one configuration file you will have to edit before starting the DHCP server:. Copy and edit the provided sample file:

The general layout of the file is set up in an indented fashion and looks like this:

Sample dhcpd.conf layout

The  block is optional and should be used for IPs you want to assign that belong to the same network topology. At least one  must be declared and the optional   block allows you to group options between items. A good example of looks like this:

Sample dhcpd.conf

The IP address after  will be asked for the specified. This IP address should be the IP of the tftp server, usually the same as the master's IP address. The  is relative to the  directory (this is due to the tftp server specific options which will be covered later). Inside the  block, the   option specifies a MAC address, and   assigns a fixed IP address to that particular MAC address. There is a pretty good man page on with options that are beyond the scope of this HOWTO. You can read it by typing:

Starting the DHCP server
Before you start the dhcp initialization script edit the file so that it looks something like this:

Sample /etc/conf.d/dhcp

The  variable is the device you wish to run your DHCP server on, in our case. Adding more arguments to the  variable can be useful for a complex network topology with multiple Ethernet cards. To start the dhcp server type:

To add the dhcp server to your start-up scripts type:

Troubleshooting the DHCP server
To see if a node boots you can take a look at. If the node successfully boots, the file should have some lines at the bottom looking like this:

Sample log file entries created by dhcp

If you get the following message it probably means there is something wrong in the configuration file but that the DHCP server is broadcasting correctly.

Sample dhpc server error

Every time you change the configuration file you must restart the DHCP server. To restart the server type:

About the TFTP server
TFTP stands for Trivial File Transfer Protocol. The TFTP server is going to supply the slaves with a kernel and an initial filesystem. All of the slave kernels and filesystems will be stored on the TFTP server, so it's probably a good idea to make the master the TFTP server.

Installing the TFTP server
A highly recommended tftp server is available as the tftp-hpa package. This tftp server happens to be written by the author of SYSLINUX and it works very well with pxelinux. To install simply type:

Configuring the TFTP server
Edit. You need to specify the tftproot directory with  and any command line options with. It should look something like this:

Sample /etc/conf.d/in.tftpd

The  option indicates that this server listens in stand alone mode so you don't have to run inetd. The  indicates that log/error messages should be verbose. The  specifies the root of your tftp server.

Starting the TFTP Server
To start the tftp server type:

This should start the tftp server with the options you specified in the. If you want this server to be automatically started at boot type:

About PXELINUX
This section is not required if you are only using Etherboot. PXELINUX is the network bootloader equivalent to LILO or GRUB and will be served via TFTP. It is essentially a tiny set of instructions that tells the client where to locate its kernel and initial filesystem and allows for various kernel options.

Before you get started
You will need to get the pxelinux.0 file which comes in the SYSLINUX package by H. Peter Anvin. You can install this package by typing:

Setting up PXELINUX
Before you start your tftp server you need to setup pxelinux. First copy the pxelinux binary into your directory:

This will create a default bootloader configuration file. The binary will look in the  directory for a file whose name is the client's IP address in hexadecimal. If it does not find that file it will remove the rightmost digit from the file name and try again until it runs out of digits. Versions 2.05 and later of syslinux first perform a search for a file named after the MAC address. If no file is found, it starts the previously mentioned discovery routine. If none is found, the file is used.

Files that PXE looks for in pxelinux.cfg/ in sequence

Let's start with the file:

Sample pxelinux.cfg/default

The  tag directs pxelinux to the kernel bzImage that we compiled earlier. The  tag appends kernel initialisation options. Since we compiled the slave kernel with , we will specify the nfsroot here. The first IP is the master's IP and the second IP is the directory that was created in to store the slave's initial filesystem.

About Etherboot
Etherboot boots network boot images from a TFTP server. As the PXE this is equivalent to LILO or GRUB. The  utility enables you to create different images using different options.

Before you get started
You will need to get the  (utility for making tagged kernel images useful for netbooting) package to create your Etherboot images. This tool will create a preconfigured kernel image from your original kernel. This contains the boot options as shown further down.

Setting up Etherboot
In this section we will create a simple etherboot image. As the dhcp server gives out the clients root-path in the "option root-path" dhcp.conf, we do not have to include this here. More details can be found in the mknbi manual.

Making the boot images. This will create a ELF bootable image capable of passing dhcp and the rootpath to the kernel. Also forcing the kernel to browse the network for a dhcp server.

Troubleshooting the network boot process
There are a few things you can do to debug the network boot process. Primarily you can use a tool called. To install  type:

Now you can listen to various network traffic and make sure your client/server interactions are functioning. If something isn't working there are a few things you might want to check. First make sure that the client/server is physically connected properly and that the networking cables are not damaged. If your client/server is not receiving requests on a particular port make sure that there is no firewall interference. To listen to interaction between two computers type:

You can also use  to listen on particular port such as the tftp port by typing:

A common error you might receive is: "PXE-E32: TFTP open time-out". This is probably due to firewall issues. If you are using , you might want to check  and  and make sure that they are configured properly. The client should be allowed to connect to the server.

About the NFS server
NFS stands for Network File System. The NFS server will be used to serve directories to the slave. This part can be somewhat personalized later, but right now all we want is a preliminary slave node to boot diskless.

About Portmapper
Various client/server services do not listen on a particular port, but instead rely on RPCs (Remote Procedure Calls). When the service is initialised it listens on a random port and then registers this port with the Portmapper utility. NFS relies on RPCs and thus requires Portmapper to be running before it is started.

Before you start
The NFS Server needs kernel level support so if you don't have this you should recompile your master's kernel. To double check your master's kernel configuration type:

You should see output that looks something like this if your kernel has been properly configured:

Installing the NFS server
The NFS package that can be acquired through portage by typing:

This package will emerge a portmapping utility, nfs server, and nfs client utilities and will automatically handle initialisation dependencies.

Configuring the NFS server
There are three major configuration files you will have to edit:

Nfs configuration files

The file specifies how, to who and what to export through NFS. The slave's fstab will be altered so that it can mount the NFS filesystems that the master is exporting.

A typical for the master should look something like this:

Sample master /etc/exports

The first field indicates the directory to be exported and the next field indicates to who and how. This field can be divided in two parts: who should be allowed to mount that particular directory, and what the mounting client can do to the filesystem:  for read only,   for read/write;   and   are important for diskless clients that are writing to the disk, so that they don't get "squashed" when making I/O requests. The slave's fstab file,, should look like this:

Sample slave fstab

In this example, master is just the hostname of the master but it could easily be the IP of the master. The first field indicates the directory to be mounted and the second field indicates where. The third field describes the filesystem and should be NFS for any NFS mounted directory. The fourth field indicates various options that will be used in the mounting process (see mount(1) for info on mount options). Some people have had difficulties with soft mount points so we made them all hard, but you should look into various options to make your cluster more efficient.

The last file you should edit is which describes a few options for nfs when it is initialised and looks like this:

Sample master /etc/conf.d/nfs

You should change  to the number of diskless nodes on the network.

Starting the NFS server
You should start the nfs server with its init script located in by typing:

If you want to this script to start when the system boots simply type:

Copy the missing files
We will now make the slave's file system in sync with the master's and provide the necessary binaries while still preserving slave specific files.

Configure diskless networking
In order to prevent the networking initscript from killing the connection to your NFS server, you will need to add an option to on your diskless client's filesystem.

Editing /etc/conf.d/net

Initialisation scripts
You need as many init scripts under as you need services on your diskless nodes. It all depends on what you want your slaves to do.

Typical slave runlevels

Now is a good time to boot your slave and cross your fingers. It works? Congratulations, you are now the proud owner of (a) diskless node(s) :)

Acknowledgements
We would like to thank the following authors and editors for their contributions to this guide:


 * Michael Andrews
 * Kristian Jerpetjoen
 * Sven Vermeulen
 * Xavier Neys