Gentoo On X-Gene 1 Mudan

Overview
The Mudan is an AARCH64 (arm64) server board based on the APM X-Gene 1 CPU. It has an a Board Management Controller (BMC) based on the ASPEED-2400 chipset.

The BMC provides remote power on/off, JavaSOL for a remote serial link and for a remote (graphical?) console viewer jviewer. These are both Java Webstart applications and offer low or zero functionality with current javaws.

Plan to do the install without a console. Mine doesn't work yet.

The idea is to prepare the boot disk, install it in the Mudan, boot, then ssh in from a safe distance.

Preparation
Its an EFI based system. Follow the AMD64 handbook for partitioning and so on. Do remember to use the [//distfiles.gentoo.org/experimental/arm64/ arm64 stage3]!!

Use grub as your boot-loader.

A bootable arm64 Linux is required. There are a few steps that must be carried out on the Mudan itself. The Debian one was used after discovering that the Fedora offering would not drive the console.

To use VNC with Fedora, follow this guide. TL;DR you will need to add the following to the boot options: inst.vnc inst.vncpassword=fedora99

Ethernet Ports
The Mudan has two 1GBit Ethernet ports. The one on top of the two USB ports is dedicated to the BMC. The one on its own, or by the two 10Gbit ports is for the server itself. Connect both of them.

Framebuffer Console
After some hints that the framebuffer console is broken sometime after the 4.14 kernel, its confirmed that gentoo-sources:4.14.188 provides a working framebuffer console. Further work shows that EFI framebuffer is broken but astdrmfb, free with the AST DRM driver, works.

The remote graphical console is supposed to work with the jviewer application but jviewer would not start for this writer

Serial Console
Its Serial Console at 115200 baud delivered over Ethernet. In the Mudan, its a real 115200 baud port, redirected. That means its as fast/slow as 115200 baud link. Its pretending to be talking to a [//en.wikipedia.org/wiki/VT100 DEC] VT100 terminal.

The JavaSOL serial console works in gentoo-sources:4.14.188. However, JavaSOL should be avoided as it keeps crashing. Especially when provoked by ncurses.

offers a better alternative in that it is much faster to restart when it locks up.

There are command line ipmi commands for all the other information and commands the remote management web interface provides too.

The Install - Off the Mudan
Do as much as possible away from the Mudan. This author chrooted in to the Mudan install on a Raspberry Pi4 to do the rc-config setup and build grub then cross compiled the kernel.

Partition The Drive Untar the Stage3 Untar the portage snapshot

The Kernel
Installing crossdev and the kernel build process is covered by Raspberry_Pi_3_64_bit_Install.

The key differences are: The kernel options required As the Mudan is an EFI system, the EFI firmware provides the Device Tree Binary (*.dtb)

The manual kernel configuration and install process is assumed to create a kernel that can boot without the aid of an initrd.

Kernel Options
This author used the Raspberry Pi kernel tree and added APM-X-Gene and AST options. This provides a system that can boot on both a Raspberry Pi and the Mudan. That's a very good thing for user space testing and configuration and is an excellent excuse to invest in a Pi 4 too.

Be aware that the kernel can be compiled to run on X-Gene, which in the main processor or on the AST-2400 BMC. The options for building the kernel to run on the BMC are hidden behind ARCH_ASPEED, which will be off and hidden by the ARCH=arm64 inherited from the environment. The AST-2400 BMC is a 32 bit arm system.

Under Platform selection [*] Broadcom BCM2835 family [*] AppliedMicro X-Gene SOC Family will do nicely.

Use the make menuconfig search to find all the symbols with XGENE in their names and enable them as <*>.

<*> AST server chips

Should be the DRM driver for AST GPU but I don't get anything on the VGA output once the kernel loads.

Install the kernel to suit both the Pi and Mundan under the stage3 and boot it on the Pi. Unless your Pi does USB booting, the kernel binary will need to be installed into both the Mudan and Pis /boot.

Grub
On the Pi, install Grub2 following Handbook:AMD64/Installation/Bootloader.

/etc/default/grub requires the following additions LANG=C GRUB_DISABLE_LINUX_UUID=true GRUB_DISABLE_LINUX_PARTUUID=false
 * 1) fix boot error    error: file /boot/grub/locale/C.gmo not found
 * 1)  Do not use UUID do use PARTUUID

Grub can be installed but as the Pi is not an EFI system, it will fail trying to to update the efivars. Repeating this step is the only thing that must be done on the Mudan.

Userspace
Adding the following is a good start.

app-admin/sudo app-misc/screen app-portage/eix app-portage/gentoolkit app-text/wgetpaste net-misc/dhcpcd net-misc/ntp sys-apps/pciutils sys-apps/usbutils sys-boot/grub sys-process/htop

Access will be via ssh. You will need a way to obtain root over ssh. Be sure that you have a way to do that before the HDD is moved to the Mudan.

Power Requirements
The test system comprises the APM X-Gene 1 motherboard, fitted with 4x16G PC-1333 RDIMMs a single ex k6-2 fan fitted to the CPU, (no case fans yet) and a single 1TB 'spinney' HDD.

Its all powered by a Coolermaster White 500w PSU. The following power measurements are AC Input Power to the PSU.

Standby 3.5w (PSU on and system powered off) Idle 63w (After booting but doing nothing) Max 90w (Building gcc on all 8 cores)

At such low loads for a 500w PSU, the power factor is terrible but that only matters for sizing a UPS. That 90w is 110 VA.

In the UK, the running cost is just under £10/month, running the system flat out 24/7

The First Boot
Boot the Debian arm64 install from a USB stick and use its Rescue Mode to get a shell from the Gentoo install. That will take a bit of coaxing from JavaSOL.

Mount /boot if its not already mounted and rerun grub-install. This time it will update the efivars.

Shutdown the system.

The First Gentoo Boot
Go into the setup and choose the gentoo entry as the first boot option. Save and Reset then wait while it does its thing. You may or may not see the grub menu.

Discover the IP address that the Mudan is on and log in over ssh.

A Bootable USB
Since I want to try root in LVM without an initrd, I've made a bootable USB stick. Its a arm64 stage3, with gentoo-sources:4.14.188, so the framebuffer console works, with a few other packages to make logging in over ssh 'just work'.

Its [provided] as a single tarball for root and /boot. There is no ::gentoo repo, distfiles or packages. emerge works if the missing parts are provided.

The root password is and the normal user, named  has 'demouser' for a password. Root password based logins over ssh are not permitted but is in the  group.

How to use it
A 4G USB stick or larger is required.

Filesystem     Size  Used Avail Use% Mounted on /dev/sdb2        13G  1.9G   11G  16% /mnt/gentoo /dev/sdb1      122M   22M  100M  19% /mnt/gentoo/boot

Make a gpt partition table on a USB stick:

Then create a vfat filesystem for /boot, and an ext4 filesystem for root. Add swap to taste but its intended as an install tool, not a working system.

That's a ext4 filesystem with a 2kB block size an inode for every block without a journal with b-tree directories only two superblock backups

Fetch the [X-gene.xz] tarball and untar it.

Finishing up
The USB stick does not need any efivars written to boot. It's almost like a floppy.

You will need to edit its /etc/fstab as that uses my UUIDs and rootfsck will fail.

grub.cfg contains root=/dev/sdb2 which may not be correct for your configuration. Either edit  or use the grub editor during boot.

None of these edits require an arm64 system.

Booting
Connect the USB stick and go into the firmware to select the stick as the first boot option. Save the change and reset. The system will restart. Let it get to the grub menu, then edit the root filesystem as required. Continue the boot.

You should have both a framebuffer and serial console. The serial console works with javaSOL.

eth0 will have started with DHCP, so once the IP address is known, its possible to use demouser@ to log in via ssh.

May Be Useful, Possibly Only Vaguely Related Things
Information on the Mudan Server is very thin on the ground. Links to things that might be useful go here.

Gigabyte MP30-AR0
Gigabyte marketed a server based on the X-Gene 1 and AST2400 BMC. The bits are scattered around on the motherboard differently to the Mudan https://www.gigabyte.com/uk/Server-Motherboard/MP30-AR0-rev-11/support#support-manual but its the same major components.

WikiChip
WikiChip has a page or two https://en.wikichip.org/wiki/apm/x-gene

There are product briefs for a 1U and 2U server.

Fitting RAM
From trial and error the Mudan is very picky about where RAM is fitted and which order RAM is fitted.

The Blue sockets must be populated first.

Its worse than that. RAM must be fitted in powers of 2 sticks. That's 1, 2, 4 or 8. Trial and error suggests that mixed RAM sizes are not tolerated but mixed speeds are.

Low voltage (1.35v) DIMMs are accepted but still operated at 1.5v, which will not harm the RAM.

Ethernet Ports
The Mudan has four. One on its own, that is for the IPMI.

One with the two SFP+ ports, which the kernel sees as eth0. None of the Ethernet ports are on the PCIe bus, so its unlikely that persistent device renaming will change the names.

Ethertool reports Settings for eth0: Supported ports: [ TP	 MII ] Supported link modes:  10baseT/Full 100baseT/Full 1000baseT/Full Settings for eth1: Supported ports: [ FIBRE ] Supported link modes:  10000baseT/Full
 * 1) ethtool eth0
 * 1) ethtool eth1

Settings for eth2: Supported ports: [ FIBRE ] Supported link modes:  10000baseT/Full
 * 1) ethtool eth2

Having tested an SFP 1Gb/sec SFP insert in eth1 and eth2 the author can confirm that the Mudans SFP+ are not backwards compatible with SFP RJ45 inserts.

A Pi 4 Chroot
With 8 cores and lots of RAM, the Mudan is an attractive build system for a Raspberry Pi binhost.

As always, there are a few problems with building on a CPU that cannot always execute the code its just built.

The X-Gene 1 is the only armv8 CPU that lacks the crc extensions. That does not appear to be a problem with core system packages as my toolchain still works.

/etc/portage/env/x-gene
We need to tell portage what the X-Gene 1 CPU really is, for those packages that need it. Avoiding Illegal instruction issues like /var/log/portage/net-libs:signond-8.60-r2:20201007-172655.log-/usr/lib64/qt5/bin/qdbusxml2cpp -a backup_adaptor.h: ../../lib/signon /com.nokia.SingleSignOn.Backup.xml /var/log/portage/net-libs:signond-8.60-r2:20201007-172655.log:make[2]: *** Illegal instruction

CFLAGS="${CFLAGS} -march=armv8-a -mtune=cortex-a72" CXXFLAGS="${CXXFLAGS} -march=armv8-a  -mtune=cortex-a72"
 * 1) revert the +crc which the X-Gene 1 does not have
 * 2) Hmm, maybe its not cortex-a72 either.
 * 3) just on a per package basis   -mcpu=armv8-a

/etc/portage/package.env/x-gene
Now tell portage to build a few packages specifically for the X-Gene 1.

dev-qt/qtdbus x-gene dev-qt/qtcore x-gene

We may still be able to build these packages for the Pi 4 with FEATURES=buildpkgonly. The trick is not in try to execute them on the X-Gene 1.

FEATURES=buildpkgonly requires that all dependencies already be installed, as portage is not permitted to install anything to the filesystem while it is in effect.