Gentoo On X-Gene 1 Mudan
- 1 Overview
- 2 Preparation
- 3 Gotchas
- 4 The Install - Off the Mudan
- 5 Move to the Mudan
- 6 A Bootable USB
- 7 May Be Useful, Possibly Only Vaguely Related Things
The Mudan is an AARCH64 (arm64) server board based on the APM X-Gene 1 CPU. It has an a Board Management Controller (BMC) based on the ASPEED-2400 chipset.
The BMC provides remote power on/off, JavaSOL for a remote serial link and for a remote (graphical?) console viewer jviewer. These are both Java Webstart applications and offer low or zero functionality with current javaws.
Plan to do the install without a console. Mine doesn't work yet.
The idea is to prepare the boot disk, install it in the Mudan, boot, then ssh in from a safe distance.
Use grub as your boot-loader.
A bootable arm64 Linux is required. There are a few steps that must be carried out on the Mudan itself. The Debian one was used
after discovering that the Fedora offering would not drive the console.
To use VNC with Fedora, follow this guide. TL;DR you will need to add the following to the boot options:
The Mudan has two 1GBit Ethernet ports. The one on top of the two USB ports is dedicated to the BMC. The one on its own, or by the two 10Gbit ports is for the server itself. Connect both of them.
After some hints that the framebuffer console is broken sometime after the 4.14 kernel, its confirmed that gentoo-sources:4.14.188 provides a working framebuffer console. Further work shows that EFI framebuffer is broken but astdrmfb, free with the AST DRM driver, works.
The remote graphical console is supposed to work with the jviewer application but jviewer would not start for this writer
Its Serial Console at 115200 baud delivered over Ethernet. In the Mudan, its a real 115200 baud port, redirected. That means its as fast/slow as 115200 baud link. Its pretending to be talking to a DEC VT100 terminal.
The JavaSOL serial console works in gentoo-sources:4.14.188. However, JavaSOL should be avoided as it keeps crashing. Especially when provoked by ncurses.
offers a better alternative in that it is much faster to restart when it locks up.
There are command line ipmi commands for all the other information and commands the remote management web interface provides too.
The Install - Off the Mudan
Do as much as possible away from the Mudan. This author chrooted in to the Mudan install on a Raspberry Pi4 to do the rc-config setup and build grub then cross compiled the kernel.
Partition The Drive Untar the Stage3 Untar the portage snapshot
Installing crossdev and the kernel build process is covered by Raspberry_Pi_3_64_bit_Install.
The key differences are:
The kernel options required As the Mudan is an EFI system, the EFI firmware provides the Device Tree Binary (*.dtb)
The manual kernel configuration and install process is assumed to create a kernel that can boot without the aid of an initrd.
This author used the Raspberry Pi kernel tree and added APM-X-Gene and AST options. This provides a system that can boot on both a Raspberry Pi and the Mudan. That's a very good thing for user space testing and configuration and is an excellent excuse to invest in a Pi 4 too.
Be aware that the kernel can be compiled to run on X-Gene, which in the main processor or on the AST-2400 BMC. The options for building the kernel to run on the BMC are hidden behind ARCH_ASPEED, which will be off and hidden by the ARCH=arm64 inherited from the environment. The AST-2400 BMC is a 32 bit arm system.
Under Platform selection
[*] Broadcom BCM2835 family [*] AppliedMicro X-Gene SOC Family
will do nicely.
Use the make menuconfig search to find all the symbols with XGENE in their names and enable them as <*>.
<*> AST server chips
Should be the DRM driver for AST GPU but I don't get anything on the VGA output once the kernel loads.
Install the kernel to suit both the Pi and Mundan under the stage3 and boot it on the Pi. Unless your Pi does USB booting, the kernel binary will need to be installed into both the Mudan and Pis /boot.
On the Pi, install Grub2 following Handbook:AMD64/Installation/Bootloader#Default:_GRUB2.
/etc/default/grub requires the following additions
# fix boot error error: file /boot/grub/locale/C.gmo not found LANG=C # Do not use UUID do use PARTUUID GRUB_DISABLE_LINUX_UUID=true GRUB_DISABLE_LINUX_PARTUUID=false
The use of the filesystem UUID requires an initrd to provide the userspace mount command.
Grub can be installed but as the Pi is not an EFI system, it will fail trying to to update the efivars. Repeating this step is the only thing that must be done on the Mudan.
Adding the following is a good start.
app-admin/sudo app-misc/screen app-portage/eix app-portage/gentoolkit app-text/wgetpaste net-misc/dhcpcd net-misc/ntp sys-apps/pciutils sys-apps/usbutils sys-boot/grub sys-process/htop
Access will be via ssh. You will need a way to obtain root over ssh. Be sure that you have a way to do that before the HDD is moved to the Mudan.
Move to the Mudan
The test system comprises the APM X-Gene 1 motherboard, fitted with 4x16G PC-1333 RDIMMs a single ex k6-2 fan fitted to the CPU, (no case fans yet) and a single 1TB 'spinney' HDD.
Its all powered by a Coolermaster White 500w PSU. The following power measurements are AC Input Power to the PSU.
Standby 3.5w (PSU on and system powered off) Idle 63w (After booting but doing nothing) Max 90w (Building gcc on all 8 cores)
At such low loads for a 500w PSU, the power factor is terrible but that only matters for sizing a UPS. That 90w is 110 VA.
In the UK, the running cost is just under £10/month, running the system flat out 24/7
The First Boot
Boot the Debian arm64 install from a USB stick and use its Rescue Mode to get a shell from the Gentoo install. That will take a bit of coaxing from JavaSOL.
Mount /boot if its not already mounted and rerun grub-install. This time it will update the efivars.
Shutdown the system.
The First Gentoo Boot
Go into the setup and choose the gentoo entry as the first boot option. Save and Reset then wait while it does its thing. You may or may not see the grub menu.
Discover the IP address that the Mudan is on and log in over ssh.
A Bootable USB
Since I want to try root in LVM without an initrd, I've made a bootable USB stick. Its a arm64 stage3, with gentoo-sources:4.14.188, so the framebuffer console works, with a few other packages to make logging in over ssh 'just work'.
Its [provided] as a single tarball for root and /boot. There is no ::gentoo repo, distfiles or packages. emerge works if the missing parts are provided.
The root password is root and the normal user, named demouser has 'demouser' for a password. Root password based logins over ssh are not permitted but demouser is in the wheel group.
How to use it
A 4G USB stick or larger is required.
The worked example uses /dev/sdb for the USB stick. Use whatever your USB stick appears as.
Filesystem Size Used Avail Use% Mounted on /dev/sdb2 13G 1.9G 11G 16% /mnt/gentoo /dev/sdb1 122M 22M 100M 19% /mnt/gentoo/boot
Make a gpt partition table on a USB stick:
Then create a vfat filesystem for /boot, and an ext4 filesystem for root. Add swap to taste but its intended as an install tool, not a working system.
mkfs -t vfat /dev/sdX1
mke2fs -t ext4 -b 2048 -i 2048 -O^has_journal,dir_index,sparse_super2 /dev/sdX2
That's a ext4 filesystem
with a 2kB block size an inode for every block without a journal with b-tree directories only two superblock backups
If your e2fsprogs is too old, you may not be able to use 'sparse_super2' and only 'sparse_super'
mount /dev/sdX2 /mnt/gentoo
mount /dev/sdX1 /mnt/gentoo/boot
Fetch the [X-gene.xz] tarball and untar it.
tar --xattrs --strip-components=2 --show-transformed-names -Jxpf <X-gene.xz> -C /mnt/gentoo/
The USB stick does not need any efivars written to boot. It's almost like a floppy.
You will need to edit its /etc/fstab as that uses my UUIDs and rootfsck will fail.
grub.cfg contains root=/dev/sdb2 which may not be correct for your configuration. Either edit /boot/grub/grub.cfg or use the grub editor during boot.
Neither rootwait nor root=PARTUUID=... work from my USB stick
None of these edits require an arm64 system.
Connect the USB stick and go into the firmware to select the stick as the first boot option. Save the change and reset. The system will restart. Let it get to the grub menu, then edit the root filesystem as required. Continue the boot.
You should have both a framebuffer and serial console. The serial console works with javaSOL.
eth0 will have started with DHCP, so once the IP address is known, its possible to use demouser@ to log in via ssh.
May Be Useful, Possibly Only Vaguely Related Things
Information on the Mudan Server is very thin on the ground. Links to things that might be useful go here.
Gigabyte marketed a server based on the X-Gene 1 and AST2400 BMC. The bits are scattered around on the motherboard differently to the Mudan https://www.gigabyte.com/uk/Server-Motherboard/MP30-AR0-rev-11/support#support-manual but its the same major components.
WikiChip has a page or two https://en.wikichip.org/wiki/apm/x-gene
There are product briefs for a 1U and 2U server.
From trial and error the Mudan is very picky about where RAM is fitted and which order RAM is fitted.
The Blue sockets must be populated first.