ESPRESSOBin

Overview
This documents the basic setup for Marvell Espressobin ARM64 network switch-ish board. It appears to be the only similar-class Marvell board with open docs (ie, not protected by NDA). Googling reveals a mix of both proprietary and "open" resources (and github appears to contain a mix of both, so ignore links and other references to "extranet"). There appears to be one main site for the ESPRESSOBin boards, with the corresponding vendor sources mainly on github.

History
The Marvell Espressobin was originally a kickstarter board by Globalscale Technologies Inc. It is a high performance 64 bit dual-core networking computing platform based on the ARMv8 architecture and has low power consumption. The board is powered by Marvell's Armada 3700 dual-core SoC chipset which runs up to 1.2GHz.

At this point there are several variants of the Espressobin hardware, however, the newer device trees exist only in the Marvell and GTI github repos (and corresponding vendor kernel images). The actual hardware differences are mainly DDR size/speed and eMMC (whether it exists and how big). The board stencil should show which hardware version is present:


 * v5 espressobin: DDR3 with or without soldered eMMC
 * v7 espressobin: DDR4 with or without soldered eMMC
 * "ultra" espressobin: DDR4 with 8 GB eMMC

The above differences are reflected in the new device-tree names.

Pros:


 * Solid mainline support
 * Great network/SATA performance
 * Mini PCIe expansion slot
 * 2 GbE LAN ports, 1 GbE WAN port
 * USB 3 host port
 * ATF-based boot protocol
 * 1 GB or 2 GB RAM

Cons:


 * no video/audio
 * ATF-based boot protocol
 * somewhat complicated u-boot build process

The Gentoo arm64 offering is still experimental at the time of writing. That means that very little is marked stable. Expect to use ACCEPT_KEYWORDS="~arm64" and then use package.accept_keywords too.

What Works
All the core hardware should be supported in latest mainline kernel (tested on 4.13.12) and u-boot 2017.11-rc4.
 * Network switch
 * USB 2.0/3
 * MMC
 * SATA
 * PCIE
 * UART over micro-USB
 * Hardware Crypto
 * SPI flash

Untested

 * GPIO/SPI/I2C (on the connectors)
 * PCIe

What's Required

 * Gentoo Install on a PC
 * microSD card reader for the PC
 * ESPRESSOBin board
 * 12V PSU
 * microSD card > 8G
 * micro USB cable
 * network cable

The contents of the microSD card will be wiped during the install.

Update U-boot
Starting with the 5.10.x mainline kernel, the U-boot/ATF firmware needs updating in order to load the newer device trees (required for eMMC and v7 hardware). The build process is reasonably well-documented on the espressobin wiki (see the Reference links at the bottom). Some of us are lazy (or just need to do other stuff) and have already tested the flash images from the espressobin wiki to make sure they load the new device trees (download said flash image blobs from the espressobin tech-spec page and flash either the debug or release blob for the board). Do not forget to setup the u-boot environment again after flashing. Also note this applies if using any of the newer vendor kernels (either Marvell or GTI).

Installation Overview

 * Install crossdev on the PC
 * Build mainline u-boot (optional)
 * Fetch the mainline kernel (or use the builder script)
 * Partition the microSD card
 * Fetch the Gentoo bits of the install
 * Cross compile and install the kernel
 * Setup networking and root passwd
 * Boot with serial console to test

Install Crossdev on the PC
crossdev is Gentoos' tool for building cross compiler tool chains. Once it's installed, it will be used to build the arm64 kernel on the Gentoo PC. will install the crossdev tool.

Build mainline u-boot (optional)
U-boot has a rather complicated build and flash process, which is now documented in the repo manifest link below. The version of u-boot in SPI flash on the board I received is 2015.01-armada-17.02.0-g8128e91 and the default environment is set to boot from mmc. The 2017 updates seem to work fine, albeit with worse initial env config than factory.

As mentioned in the above note, building u-boot (either mainline or Marvell) is optional; that said, some extra "features" can be gained by building/flashing mainline u-boot instead of the Marvell fork. One useful feature in mainline that's already enabled are the distro defaults, which includes the extlinux/syslinux boot config file.

To build mainline u-boot/ATF for espressobin, either follow the manual steps in the two reference links below, or else follow the readme steps on the marvell-armada branch of this repo manifest

After building the aarch64 toolchain, the secondary toolchain can be built with a command something like this:

Use Crossdev to Build A Cross Compiler
Recommended crossdev setup:


 * do use the most current version, eg, at least sys-devel/crossdev-20201129
 * do setup your crossdev overlay first
 * optionally adjust USE and EXTRA_ECONF flags as needed

To setup an ebuild repository for crossdev to use, see the appropriate section in the Crossdev article. Setup your overlay as shown, populate the config files, then use the --ov-output  option to set the path to your overlay.

The crossdev command expects at least the target tuple, with optional version specifiers for the toolchain components and more (see the help output and README file). The short command above will use the "latest" versions available and its own overlay detection (which you probably don't want) so use of the output-overlay argument is recommended, as shown in the following examples.

There are some crossdev examples in the arm overlay; an example "fancy" crossdev command for arm64 might be:

Convert files as crossdev asks e.g. error: please convert /etc/portage/package.env to a directory by appending _file to the existing filename

making the directory

then moving package.env_file into the directory.

Rinse and repeat until crossdev is happy.

crossdev will take a while. It is building binutils:             binutils-[latest] gcc:                  gcc-[latest] headers:              linux-headers-[latest] libc:                 glibc-[latest]

When crossdev completes, the cross toolchain is ready:

It will also create an arm64 target root in This is used by cross emerge.

Pure cross compiling, other than the kernel, is out of scope of this guide.

Fetch, Configure and Build the Kernel
There are some prebuilt vendor kernel images for this board, but mainline is preferred.

Fetch the Kernel
The kernel choices, either "stock" mainline or gentoo-sources, can be built by hand, or using the kernel build scripts with some Armbian patches (older kernels only). Note there are older ebuilds in the arm overlay that incorporate the appropriate gentoo patches, mvebu64 patches, as well as updated defconfigs available, see https://github.com/sarnold/arm64-multiplatform

UPDATE: For the Gentoo way with no extra patches, use gentoo-sources 5.10.14-ish *and* the mainline u-boot flash-image.

So either use the builder scripts (borrowed from RCN repo and updated for ESPRESSOBin) or grab the defconfig and do it manually. The build scripts will clone linux-stable if not configured (see the file system.sh.sample and copy it to system.sh, then set your CC prefix kernel path).

For the latest, use the v5.10.x branch, edit system.sh, then run build_kernel.sh and wait for your shiny new kernel in the deploy/ directory.

At the end of the build, it will output the kernel version string to export for the install commands later. Something like:

- Script Complete eewiki.net: [user@localhost:~$ export kernel_version=4.13.12-aarch64-r0] -

Configure The Kernel
Once the scripts have cloned the kernel tree and applied any patches, it will wait at the menuconfig screen. Note the default config used should have all the required hardware support already enabled, but may not have all the network tools you want yet (the config is still a WIP).

More confident readers may be tempted to trim things out or add things now. A word of advice - don't, at least, not until the system boots.

Cross Compiling The Kernel
The scripts will use the toolchain arch set in system.sh but if you want to compile it manually the proper command line would be something like:

Background
The current vendor u-boot environment is already set to look for a kernel and device tree blob in the boot directory of the first partition, as well as use the first partition as root. Therefore only one partition is required (or at least make sdX1 the rootfs).

Partitioning
Depending on how the microSD card is connected to you PC, it my be /dev/sdX or /dev/mmcblkY

In the example below, it's /dev/sdb. On sdcards, using a swapfile instead of a separate partition is recommended.

Clean the card and format with options
Using the partitioning tool of your choice, make one partition on your microSD card. root (/)

Using fdisk, and your microSD card block device, not the example /dev/sdb unless it's correct for your setup

Use ext4 for root:

The i-node count cannot be changed after filesystem creation and can limit the number of files on a files system. The Gentoo repository alone needs over 17,000 i-nodes.

Fetch the Gentoo bits of the install
To make it easy to cross refer to the Gentoo_Handbook

Mount the microSD card root filesystem at /mnt/gentoo

Install the arm64 Stage 3
Following the Gentoo_Handbook fetch the arm64 stage3 and untar it to /mnt/gentoo in the normal way.

/mnt/gentoo/tmp should be empty. Clear it now.

Install a Portage Snapshot
This step is not actually needed to boot but emerge won't work without it

Following the Gentoo Handbook, fetch and unpack a portage snapshot in the normal way.

Careful readers can copy their host.

Populating /boot
All that's needed here is the kernel binary and the proper device tree blob(s). If you have not flashed mainline u-boot, the file names have to match what the vendor u-boot expects, so just copy the files as-is and make symlinks to the required names (see below). For mainline u-boot simply use for the kernel/dtbs/modules or copy them however you wish.

Install the Kernel to the microSD Card
The kernel was built above, now to install it.

The kernel is in three parts
 * 1) kernel binary
 * 2) device tree blob
 * 3) kernel modules

Install The Kernel Binary
As root or via sudo, copy the kernel binary from the build location, either from your kernel tree (/path/to/your/tree/linux/arch/arm64/boot/Image) or the deploy directory:

Install The Device Tree
The device tree binary (.dtb) describes the hardware to the kernel. This avoids having all the existing hardware configurations hard coded into the kernel.

Extract the following .dtb file from ./arm64-multiplatform/deploy/${kernel_version}-aarch64-r0-dtbs.tar.gz and copy it to the boot directory:

Create the Symlinks for Vendor U-Boot
Change to the mounted boot directory and symlink the files to the expected names:

Install The Kernel Modules
For a hand-built/installed kernel, don't foget to run make with the modules_install arg. From the top of the kernel tree, install the kernel modules

Otherwise, unpack the kernel modules tarball from the deploy directory:

Checking The Kernel Install (vendor u-boot)
Shows that the kernel and modules were installed to the correct locations.

Flashing and Configuring the Bootloader
You really should update the bootloader firmware, especially if you still have a 2015 version in SPI flash. That said, there is some risk involved, so make sure you have solid power/serial connections. And don't panic... There are documented recovery methods if somehow your cat eats the power cord at just the wrong time.

Obtaining the Firmware
The available choices are:


 * Download the espressobin flash images
 * Build Marvell/GTI u-boot/ATF
 * Build mainline u-boot/ATF

Note the main differences between the above two build options are 1) mainline u-boot, and 2) master branches for the dependencies (as recommended in the ATF docs).

Flashing the Firmware
Once you have a shiny new flash image, use either tftp or a USB stick/sdcard to flash it. See the bootloader update page on the espressobin wiki and choose a method.

When the flash process is complete, you must complete the following steps for either vendor or mainline u-boot in order to get things booting again. The vendor upgrade images do not restore mmc or sata boot configs, and mainline u-boot has a minor bug in the default distro boot environment.

When the bubt command completes, reset the board, and then stop the autoboot process to get a u-boot prompt again. Run the following command and then hit a key to stop the boot:

When back at the u-boot prompt, load the default environment and print it.

The output from the printenv command will depend on the choice of u-boots; see the Boot from SATA section below for the vendor output.

Post-flash Configuration
A correct u-boot environment is required; choose one of the following.

Vendor u-boot
If not using mainline u-boot, the correct values must be defined for mmcroot, etc, and then saved using.

For vendor u-boot, compare the output of printenv with the vendor environment variables in the Boot from SATA section, then restore any missing ones or define what you need for root partitions, etc. There is not really any set standard, just make sure the result fulfills the basic reqs, ie the result should be able to execute the following to boot from the sdcard device:

setenv console console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 setenv bootargs $console root=/dev/mmcblk0p1 rw rootwait net.ifnames=0 biosdevname=0 mmc dev 0 ext4load mmc 0:1 $kernel_addr $image_name ext4load mmc 0:1 $fdt_addr $fdt_name booti $kernel_addr - $fdt_addr

For the initrd case, add one more ext4load command to the above; then replace the "-" with $ramdisk_addr_r in the last line. After defining the name of the initrd file, the initrd load command should look something like this:

ext4load mmc 0:1 $ramdisk_addr_r $initrd_name

Mainline u-boot
And now for something completely different... mainline u-boot. After flashing mainline u-boot, the printenv output should look like the following:

If the scriptaddr variable is not defined in the above output, then it will need to be set and saved manually. Note this variable is defined correctly in the header file, but somehow gets dropped from the default env (and is one of the required distro vars).

Run the following two commands to correct the default env:

Create extlinux.conf
The basic requirement for the distro_bootcmd to run is the extlinux.conf file; a basic example would look something like the following:

Create a file similar to the above under. Be sure to use tabs for indenting:

Marvell ESPRESSOBin Peripherals
Now that the base operating system is in place, you will need to do some file configuration by hand to get the peripherals working.

Serial Port Configuration
The Gentoo Stage3 comes with the default Gentoo serial port configuration, however, you need to make sure the default serial ports are commented out, and set the board-specific serial port.

Open up -

Make sure the ttyS serial port lines are commented, then find this line at the bottom and set the speed and tty port device -

Save and exit the file. This ensures Gentoo will launch a login getty on the correct serial interface.

Network Setup
It seems like all network switch setups are unique, but there is an effort to converge using the Distributed Switch Architecture (DSA). The ESPRESSOBin has two LAN ports and one WAN port, which apparently need a virtual "eth0" to be configured correctly. This means eth0 must be "up" (but not configured) before the actual network devices are usable. Unless you make udev naming rules, the default interfaces are lan0, lan1, and wan. Upstream docs name the left-most port as "lan1" which is where you want to plug in for local use.

The corresponding Gentoo network config needs all of these enabled, however, only lan1 is needed for initial setup. Here is the initial config for one network interface:

Root Password
There are several ways to generate a password hash for /etc/shadow; usually it suffices to copy the hash from another system.

All stage3 root filesystems should use an /etc/shadow root entry root:x::0:99999:7:::

Instead of copying the hash, you can also use openssl to generate a fresh one:


 * edit so root can login:
 * grab hash output, edit, and put here:

The password can always be changed once you are logged in.

/etc/fstab
On this board, the microSD card will be with one partition,

Edit /mnt/gentoo/etc/fstab to match.

Boot the Board to Test
Unmount the microSD card.

When the prompt returns, move the microSD card to the board, plug in micro-usb and power cables and open a console using your favorite tool, then plug in the AC side and power it up.

Log in at the serial console as root. Nothing was added to any runlevels during the install, so networking was not started, nor anything that depends on networking, like ntpd and sshd.

Many embedded boards do not have a hardware real time clock. The time will probably be Jan 1, 1970; note you can use busybox-ntpd to set the clock after the network is up.

What Next
As always with Gentoo, if it booted, that's the hard bit done.


 * 1) All The setup steps in the Gentoo Handbook
 * 2) Fix the MAC address or use a static IP
 * 3) Allow root logins via ssh
 * 4) Add a crond, a logger and other things the handbook does before the reboot.
 * 5) Add Kernel Sources (or at least the .config)

Boot from SATA
Yes, it's actually fairly easy to boot directly from an SSD connected to the SATA controller without using the micro-SDCard at all. For this to work, setup the SSD the same way as the card (hint: you can just rsync everything on the card to your SSD). Then make the u-boot changes shown below.

The default u-boot environment has several variables set for NFS and MMC card, but the key one is "bootcmd" which defaults to the card slot (note it may point "root" to the first or second partition depending on the vendor configuration. Googling will bring up the vendor page with how to boot a bricked device from SATA and restore the u-boot in SPI flash; ignore that (unless your board is really bricked).

So, switching jumpers simply to boot from the SATA device is not needed, just two modified u-boot environment variables will do it. To make this change, just pop out the SDCard and connect a serial console as above, then power it up and hit a key to stop autoboot to get a u-boot prompt.

Booting Trusted Firmware BL1: v1.2(release):armada-17.02.0: BL1: Built : 09:41:56, Jun 2 2NOTICE:  BL2: v1.2(release):armada-17.02.0: NOTICE: BL2: Built : 09:41:57, Jun  2 20NOTICE:  BL31: v1.2(release):armada-17.02.0: NOTICE: BL31: U-Boot 2015.01-armada-17.02.0-g8128e91 (Jun 02 2017 - 09:41:51) I2C:  ready DRAM: 1 GiB Board: DB-88F3720-ESPRESSOBin CPU   @ 1000 [MHz] L2    @ 800 [MHz] TClock @ 200 [MHz] DDR   @ 800 [MHz] Comphy-0: PEX0         2.5 Gbps Comphy-1: USB3         5 Gbps Comphy-2: SATA0        5 Gbps Now running in RAM - U-Boot at: 3ff2b000 U-Boot DT blob at : 000000003fa18168 MMC:  XENON-SDHCI: 0 SF: Detected W25Q32DW with page size 256 Bytes, erase size 4 KiB, total 4 MiB PCIE-0: Link down SCSI: Target spinup took 0 ms. AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode flags: ncq led only pmp fbss pio slum part sxs Net:  neta0 Hit any key to stop autoboot: 3  0 Marvell>>

Once you see the prompt above you can try some commands like "printenv" and "help". For example:

The variables we need to modify to switch defaults from SDCard partition 1 to SATA partition 1 are "bootargs" and "bootcmd". To change them, we need to run "setenv FOO" for each one, but that will only change them temporarily (suitable for testing). To save the changes to flash, we'll need to run the saveenv" command once we're happy with the changes.

Change the required variables:

Test the changes

If it works, great! If not, check the console output for errors, look for typos, etc. Once you've verified the changes do what you want, save them:

WiFi and Bluetooth
Options are mainly PCIe or USB, so take your pick and report back.

CFLAGS
This works quite well, except for the packages where it doesn't work...

Note gcc-6.x allows the use of -march=native but that will prevent the use of distcc. The above is the same as gcc-6.3 would set for -march=native anyway.

ACCEPT_KEYWORDS
Outside of the @system set, arm64 is either testing or keyword masked. Set

and expect to use package.accept_keywords too.

Upgrade gcc then rebuild all of the installed C++ software.

MAKEOPTS
With only 1G RAM, and two cores, the conventional

could be a bit aggressive for building larger things. It will force swapping or even appear to lock up completely, to the point where it won't even respond to the console. Unlike many other boards that may segfault under heavy load, this board will try to swap everything out (and hasn't locked up yet).

Use files in and entries in to set MAKEOPTS on a per package basis.

Networking
The basic config given above works fine for "typical" building and testing; if you want to make a firewall/gateway/router, that's beyond the scope of this HowTo.

sshd
The default configuration for sshd will not allow password based root logins. add your ssh public key for root make a normal user in the wheel group edit to allow password based root logins

Updating The Tool Chain
Once you boot, you may have the desire to update @world first thing. However, as of the time of this writing, the latest stage3 for arm64 was built in December 2016. A lot of things in the tool chain will be out of date with what is on the current Portage tree. Once you've booted the board and confirmed that you have an internet connection, you'll want to first run emerge --sync to get the absolute latest tree, then run perl-cleaner --all to get all of your Perl packages up to date.

Network Time Sync
This board does not have a hardware real time clock on board. There are vendors online where you can order RTC modules made for the standard interfaces, but if you don't plan to run one, I highly recommend installing a NTP client.

First, set the initial time using the 'date' command. Date and time will be entered in mmddhhmmyyyy format and the time is in 24-hr format -

As an example, if the time is 10:05PM on 7/31/2017 -

As with most things Gentoo, the NTP daemon is just an emerge away -

Remove the hardware clock service hwclock from the boot runlevel and replace it with the software clock service swclock -

Make sure you have the correct time zone set to the area which most closely matches your locale in /usr/share/zoneinfo -

As an example, if you live in California, you would do -

Install your timezone libraries -

Start the NTP client and add it to the default runlevel -

Where to Get Help
On Internet Relay Chat

On the Gentoo Forums, start a new topic in the Gentoo on ARM forum.

Acknowledgements
Everyone contributing to the arm64 software base.

Especially everyone...