ESPRESSOBin

Overview
This documents the basic setup for Marvell Espressobin ARM64 network switch-ish board. It appears to be the only similar-class Marvell board with open docs (ie, not protected by NDA). Googling reveals a mix of both proprietary and "open" resources (and github appears to contain a mix of both, so ignore links and other references to "extranet"). There appears to be one main site for the ESPRESSOBin boards, with the corresponding vendor sources mainly on github.

History
The Marvell Espressobin was originally a kickstarter board by Globalscale Technologies Inc. It is a high performance 64 bit dual-core networking computing platform based on the ARMv8 architecture and has low power consumption. The board is powered by Marvell's Armada 3700 dual-core SoC chipset which runs up to 1.2GHz.

At this point there are several variants of the Espressobin hardware, however, the newer device trees exist only in the Marvell and GTI github repos (and corresponding vendor kernel images). The actual hardware differences are mainly DDR size/speed and eMMC (whether it exists and how big). The board stencil should show which hardware version you have:


 * v5 espressobin: DDR3 with or without soldered eMMC
 * v7 espressobin: DDR4 with or without soldered eMMC
 * "ultra" espressobin: DDR4 with 8 GB eMMC

The above differences are reflected in the new device-tree names.

Pros:


 * Solid mainline support
 * Great network/SATA performance
 * Mini PCIe expansion slot
 * 2 GbE LAN ports, 1 GbE WAN port
 * USB 3 host port
 * ATF-based boot protocol
 * 1 GB or 2 GB RAM

Cons:


 * no video/audio
 * ATF-based boot protocol
 * somewhat complicated u-boot build process

The Gentoo arm64 offering is still experimental at the time of writing. That means that very little is marked stable. Expect to use ACCEPT_KEYWORDS="~arm64" and then use package.accept_keywords too.

What Works
All the core hardware should be supported in latest mainline kernel (tested on 4.13.12) and u-boot 2017.11-rc4. Network switch USB 2.0/3 MMC SATA PCIE UART over micro-USB Hardware Crypto SPI flash

Untested
GPIO/SPI/I2C (on the connectors) PCIe Mainline u-boot Note: u-boot lives in SPI flash

What's Required
Gentoo Install on a PC microSD card reader for the PC ESPRESSOBin board 12V PSU microSD card > 8G micro USB cable network cable

The contents of the microSD card will be wiped during the install.

Update U-boot
Starting with the 5.10.x mainline kernel, you will need to update the U-boot/ATF firmware in order to load the newer device trees (required for eMMC and v7 hardware). The build process is reasonably well-documented on the espressobin wiki (see the Reference links at the bottom). Some of us are lazy (or just need to do other stuff) and have already tested the flash images from the espressobin wiki to make sure they load the new device trees (you can download said flash image blobs from the espressobin tech-spec page and flash either the debug or release blob for your board). Note you will need to setup your u-boot environment again after flashing. Also note this also applies if using any of the newer vendor kernels (either Marvell or GTI).

Installation Overview

 * Install crossdev on the PC
 * Build mainline u-boot (optional)
 * Fetch the mainline kernel (or use the builder script)
 * Partition the microSD card
 * Fetch the Gentoo bits of the install
 * Cross compile and install your kernel
 * Setup networking and root passwd
 * Boot with serial console to test

Install Crossdev on the PC
crossdev is Gentoos' tool for building cross compiler tool chains. Once it's installed, it will br used to build the arm64 kernel on the Gentoo PC. will install the crossdev tool.

Build mainline u-boot (optional)
U-boot has a rather complicated build and flash process, which is now documented in the repo manifest link above. The version of u-boot in SPI flash on the board I received is 2015.01-armada-17.02.0-g8128e91 and the default environment is set to boot from mmc. The 2017 updates seem to work fine, albeit with worse initial env config than factory.

As mentioned in the above note, building u-boot (either mainline or Marvell) is optional; that said, you can gain some "features" by building/flashing mainline u-boot instead of the Marvell fork. One useful feature in mainline that's already enabled is the distro defaults, which includes the extlinux/syslinux boot config file.

If you wish to build mainline u-boot/ATF for espressobin, you can either follow the manual steps in the two reference links below, or you can follow the readme steps on the marvell-armada branch of this repo manifest

After building the aarch64 toolchain, the secondary toolchain can be built with a command something like this:

Use Crossdev to Build A Cross Compiler
You may need to disable existing overlays first; at least crossdev-99999999 gets a little confused by overlays right now, and it does need its output overlay to be the primary one. After disabling your local overlays you should run crossdev with PORTDIR_OVERLAY set to where you want the crossdev output to go.

There are other parameters you can pass to crossdev too.

You can find some crossdev examples in the arm overlay; an example "fancy" crossdev command for arm64 might be:

Convert files as crossdev asks e.g. error: please convert /etc/portage/package.env to a directory by appending _file to the existing filename

making the directory

then moving package.env_file into the directory.

Rinse and repeat until crossdev is happy.

crossdev will take a while. Its building binutils:             binutils-[latest] gcc:                  gcc-[latest] headers:              linux-headers-[latest] libc:                 glibc-[latest]

When crossdev completes you will have a cross toolchain

It will also create an arm64 target root in /usr/aarch64-unknown-linux-gnu/ This is used by cross emerge.

Pure cross compiling, other than the kernel, is out of scope of this guide.

Fetch, Configure and Build the Kernel
There are some prebuilt vendor kernel images for this board, but mainline is preferred.

Fetch the Kernel
In this case you have a choice to build "stock" mainline or gentoo-sources by hand, or use the kernel build scripts with some Armbian patches (older kernels only). Note there are older ebuilds in the arm overlay that incorporates the appropriate gentoo patches, mvebu64 patches, there are updated defconfigs available, see https://github.com/sarnold/arm64-multiplatform

UPDATE: For the Gentoo way with no extra patches, use gentoo-sources 5.10.14-ish *and* the mainline u-boot flash-image.

So either use the builder scripts (borrowed from RCN repo and updated for ESPRESSOBin) or grab the defconfig and do it manually. If you don't already have a linux-stable tree the scripts will clone it for you (see the file system.sh.sample and copy it to system.sh, then set your CC prefix).

For the latest, use the v5.10.x branch, edit system.sh, then run build_kernel.sh and wait for your shiny new kernel in the dep[loy/ directory.

At the end of the build, it will output the kernel version string to export in your shell for the install commands later. Something like:

- Script Complete eewiki.net: [user@localhost:~$ export kernel_version=4.13.12-aarch64-r0] -

Configure The Kernel
Once the scripts have cloned the kernel tree and applied any patches, you will see the menuconfig screen. Note the default config used should have all the required hardware support already enabled, but may not have all the network tools you want yet (the config is still a WIP).

More confident readers may be tempted to trim things out or add things now. A word of advice - don't, at least, not until the system boots.

Cross Compiling The Kernel
The scripts will use the toolchain arch set in system.sh but if you want to compile it manually the proper command line would be something like:

Background
The current vendor u-boot environment is already set to look for a kernel and device tree blob in the boot directory of the first partition, as well as use the first partition as root. Therefor you only need to make one partition (or at least make sdX1 the rootfs).

Partitioning
Depending on how the microSD card is connected to you PC, it my be /dev/sdX or /dev/mmcblkY

In the example below, it's /dev/sdb. Also, you can use a swapfile instead of a separate partition.

Clean the card and format with options
Using the partitioning tool of your choice, make one partition on your microSD card. root (/)

Using fdisk, and your microSD card block device, not the example /dev/sdb unless it's correct for your setup

Use ext4 for root:

The i-node count cannot be changed after filesystem creation and can limit the number of files on a files system. The Gentoo repository alone needs over 17,000 i-nodes.

Fetch the Gentoo bits of the install
To make it easy to cross refer to the Gentoo_Handbook

Mount the microSD card root filesystem at /mnt/gentoo

Install the arm64 Stage 3
Following the Gentoo_Handbook fetch the arm64 stage3 and untar it to /mnt/gentoo in the normal way.

/mnt/gentoo/tmp should be empty. Clear it now.

Install a Portage Snapshot
This step is not actually needed to boot but emerge won't work without it

Following the Gentoo Handbook, fetch and unpack a portage snapshot in the normal way.

Careful readers can copy their host /var/db/repos/gentoo.

Populating /boot
All that's needed here is the kernel binary and the proper device tree blob. Note the file names have to match what the vendor u-boot expects, so just copy the files as-is and make symlinks to the required names (see below).

Install the Kernel to the microSD Card
The kernel was built above, now to install it.

The kernel is in three parts kernel binary device tree blob kernel modules

Install The Kernel Binary
As root or via sudo, copy the kernel binary from the build location, either from your kernel tree (/path/to/your/tree/linux/arch/arm64/boot/Image) or the deploy directory:

Install The Device Tree
The device tree binary (.dtb) describes the hardware to the kernel. This avoids having all the existing hardware configurations hard coded into the kernel.

Extract the following .dtb file from ./arm64-multiplatform/deploy/${kernel_version}-aarch64-r0-dtbs.tar.gz and copy it to the boot directory:

Create the Symlinks for Vendor U-Boot
Change to the mounted boot directory and symlink the files to the expected names:

Install The Kernel Modules
If you built your own kernel by hand, you'll want to run the modules_install arg. From the top of the kernel tree, install the kernel modules

Otherwise, unpack the kernel modules tarball from the deploy directory:

Checking The Kernel Install
Shows that the kernel and modules were installed to the correct locations.

Marvell ESPRESSOBin Peripherals
Now that the base operating system is in place, you will need to do some file configuration by hand to get the peripherals working.

Serial Port Configuration
The Gentoo Stage3 comes with the default Gentoo serial port configuration, however, you need to make sure the default serial ports are commnted out, and set the board-specific serial port.

Open up /etc/inittab -

Make sure the ttyS serial port lines are commented, then find this line at the bottom and set the speed and tty port device -

Save and exit the file. This ensures Gentoo will launch a login getty on the correct serial interface.

Network Setup
It seems like all network switch setups are unique, but there is an effort to converge using the Distributed Switch Architecture (DSA). The ESPRESSOBin has two LAN ports and one WAN port, which apparently need a virtual "eth0" to be configured correctly. This means eth0 must be "up" (but not configured) before the actual network devices are usable. Unless you make udev naming rules, the default interfaces are lan0, lan1, and wan. Upstream docs name the left-most port as "lan1" which is where you want to plug in for local use.

bond0: flags=5122 mtu 1500 ether a6:e9:9a:6c:17:e1 txqueuelen 1000  (Ethernet) RX packets 0 bytes 0 (0.0 B)        RX errors 0  dropped 0  overruns 0  frame 0 TX packets 0 bytes 0 (0.0 B)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

dummy0: flags=130 mtu 1500 ether d6:73:f5:c7:65:2c txqueuelen 1000  (Ethernet) RX packets 0 bytes 0 (0.0 B)        RX errors 0  dropped 0  overruns 0  frame 0 TX packets 0 bytes 0 (0.0 B)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163 mtu 1500 inet6 fe80::f2ad:4eff:fe03:861b prefixlen 64  scopeid 0x20 ether f0:ad:4e:03:86:1b txqueuelen 532  (Ethernet) RX packets 203950 bytes 145809090 (139.0 MiB) RX errors 0 dropped 0  overruns 0  frame 0 TX packets 145454 bytes 42938767 (40.9 MiB) TX errors 0 dropped 0 overruns 0  carrier 0  collisions 0 device interrupt 8

lan0: flags=4098 mtu 1500 ether f0:ad:4e:03:86:1b txqueuelen 1000  (Ethernet) RX packets 0 bytes 0 (0.0 B)        RX errors 0  dropped 0  overruns 0  frame 0 TX packets 0 bytes 0 (0.0 B)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lan1: flags=4163 mtu 1500 inet 192.168.10.29 netmask 255.255.255.0  broadcast 192.168.10.255 inet6 fe80::f2ad:4eff:fe03:861b prefixlen 64  scopeid 0x20 ether f0:ad:4e:03:86:1b txqueuelen 1000  (Ethernet) RX packets 203950 bytes 141322190 (134.7 MiB) RX errors 0 dropped 15488  overruns 0  frame 0 TX packets 145480 bytes 41953390 (40.0 MiB) TX errors 0 dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73 mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128  scopeid 0x10 loop txqueuelen 1000  (Local Loopback) RX packets 41 bytes 2720 (2.6 KiB) RX errors 0 dropped 0  overruns 0  frame 0 TX packets 41 bytes 2720 (2.6 KiB) TX errors 0 dropped 0 overruns 0  carrier 0  collisions 0

wan: flags=4098 mtu 1500 ether f0:ad:4e:03:86:1b txqueuelen 1000  (Ethernet) RX packets 0 bytes 0 (0.0 B)        RX errors 0  dropped 0  overruns 0  frame 0 TX packets 0 bytes 0 (0.0 B)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The corresponding Gentoo network config needs all of these enabled, however, only lan1 is needed for initial setup. Here is the initial config for one network interface:


 * 1) This blank configuration will automatically use DHCP for any net.*
 * 2) scripts in /etc/init.d.  To create a more complete configuration,
 * 3) please review /usr/share/doc/openrc*/net.example* and save your configuration
 * 4) in /etc/conf.d/net (this file :]!).

dns_domain_lo="your.domain"

config_eth0=null

config_lan1="192.168.10.29 netmask 255.255.255.0 brd 192.168.10.255" routes_lan1="default via 192.168.10.1"

dns_domain_lan1="your.domain" dns_servers_lan1="8.8.8.8"

rc_net_lan1_need="net.eth0"


 * 1) The network scripts are now part of net-misc/netifrc
 * 2) In order to avoid sys-apps/openrc-0.12 from removing
 * 3) this file, this comment was
 * 4) added; you can safely remove this comment.  Please see
 * 5) /usr/share/doc/netifrc*/README* for more information.

Root Password
There are several ways to generate a password hash for /etc/shadow; usually it suffices to copy the hash from another system.

All stage3 root filesystems should use an /etc/shadow root entry root:x::0:99999:7:::

Instead of copying the hash, you can also use openssl to generate a fresh one:


 * edit so root can login:
 * grab hash output, edit, and put here:

The password can always be changed once you are logged in.

/etc/fstab
On this board, the microSD card will be /dev/mmcblk0 with one partition, /dev/mmcblk0p1

Edit /mnt/gentoo/etc/fstab to match.

/swapfile              none            swap            sw              0 0 /dev/mmcblk0p1         /               ext4            noatime         0 1

Boot the Board to Test
Unmount the microSD card.

When the prompt returns, move the microSD card to the board, plug in micro-usb and power cables and open a console using your favorite tool, then plug in the AC side and power it up.

Log in at the serial console as root. Nothing was added to any runlevels during the install, so networking was not started, nor anything that depends on networking, like ntpd and sshd.

Many embedded boards do not have a hardware real time clock. The time will probably be Jan 1, 1970; note you can use busybox-ntpd to set the clock after the network is up.

What Next
As always with Gentoo, if it booted, that's the hard bit done.

All The setup steps in the Gentoo Handbook Fix the MAC address or use a static IP Allow root logins via ssh Add a crond, a logger and other things the handbook does before the reboot. Add Kernel Sources (or at least the .config)

Boot from SATA
Yes, it's actually fairly easy to boot directly from an SSD connected to the SATA controller without using the micro-SDCard at all. For this to work, setup the SSD the same way as the card (hint: you can just rsync everything on the card to your SSD). The make the u-boot changes shown below.

The default u-boot environment has several variables set for NFS and MMC card, but the key one is "bootcmd" which defaults to the card slot (note it may point "root" to the first or second partition depending on the vendor configuration. Googling will bring up the vendor page with how to boot a bricked device from SATA and restore the u-boot in SPI flash; ignore that (unless your board is really bricked).

So, switching jumpers simply to boot from the SATA device is not needed, just two modified u-boot environment variables will do it. To make this change, just pop out the SDCard and connect a serial console as above, then power it up and hit a key to stop autoboot to get a u-boot prompt.

Booting Trusted Firmware BL1: v1.2(release):armada-17.02.0: BL1: Built : 09:41:56, Jun 2 2NOTICE:  BL2: v1.2(release):armada-17.02.0: NOTICE: BL2: Built : 09:41:57, Jun  2 20NOTICE:  BL31: v1.2(release):armada-17.02.0: NOTICE: BL31: U-Boot 2015.01-armada-17.02.0-g8128e91 (Jun 02 2017 - 09:41:51) I2C:  ready DRAM: 1 GiB Board: DB-88F3720-ESPRESSOBin CPU   @ 1000 [MHz] L2    @ 800 [MHz] TClock @ 200 [MHz] DDR   @ 800 [MHz] Comphy-0: PEX0         2.5 Gbps Comphy-1: USB3         5 Gbps Comphy-2: SATA0        5 Gbps Now running in RAM - U-Boot at: 3ff2b000 U-Boot DT blob at : 000000003fa18168 MMC:  XENON-SDHCI: 0 SF: Detected W25Q32DW with page size 256 Bytes, erase size 4 KiB, total 4 MiB PCIE-0: Link down SCSI: Target spinup took 0 ms. AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode flags: ncq led only pmp fbss pio slum part sxs Net:  neta0 Hit any key to stop autoboot: 3  0 Marvell>>

Once you see the prompt above you can try some commands like "printenv" and "help". For example:

The variables we need to modify to switch defaults from SDCard partition 1 to SATA partition 1 are "bootargs" and "bootcmd". To change them, we need to run "setenv FOO" for each one, but that will only change them temporarily (suitable for testing). To save the changes to flash, we'll need to run the saveenv" command once we're happy with the changes.

Change the required variables:

Test the changes

If it works, great! If not, check the console output for errors, look for typos, etc. Once you've verified the changes do what you want, save them:

WiFi and Bluetooth
Options are mainly PCIe or USB, so take your pick and report back.

CFLAGS
This works quite well, except for the packages where it doesn't work...

LINK_OPTS="-flto=2" VEC_OPTS="-ftree-vectorize -ftree-loop-distribution -fvect-cost-model=cheap" BASE_TUNE_OPTS="-march=armv8-a -mcpu=cortex-a53+simd" TUNE_OPTS="-march=armv8-a+crc+fp+simd -mabi=lp64 -mcpu=cortex-a53+crc+fp+simd"

CFLAGS="${TUNE_OPTS} -O2 -pipe ${VEC_OPTS} ${LINK_OPTS}" CXXFLAGS="${CFLAGS}" LDFLAGS="${CFLAGS} -fuse-linker-plugin"
 * 1) CFLAGS="${BASE_TUNE_OPTS} -O2 -pipe"

CPU_FLAGS_ARM="edsp neon thumb vfp vfpv3 vfpv4 vfp-d32 aes sha1 sha2 crc32 v4 v5 v6 v7 v8 thumb2"

Note gcc-6.x allows the use of -march=native but that will prevent the use of distcc. The above is the same as gcc-6.3 would set for -march=native anyway.

ACCEPT_KEYWORDS
Outside of the @system set, arm64 is either testing or keyword masked. Set ACCEPT_KEYWORDS="~arm64" and expect to use package.accept_keywords too.

Upgrade gcc then rebuild all of the installed C++ software.

MAKEOPTS
With only 1G RAM, and two cores, the conventional MAKEOPTS="-j3" could be a bit aggressive for building larger things. It will force swapping or even appear to lock up completely, to the point where it won't even respond to the console. Unlike many other boards that may segfault under heavy load, this board will try to swap everything out (and hasn't locked up yet).

Use files in /etc/portage/env/ and entries in /etc/portage/package.env to set MAKEOPTS on a per package basis.

Networking
The basic config given above works fine for "typical" building and testing; if you want to make a firewall/gateway/router, that's beyond the scope of this HowTo.

sshd
The default configuration for sshd will not allow password based root logins. add your ssh public key for root make a normal user in the wheel group edit /etc/ssh/sshd_config to allow password based root logins

Updating The Tool Chain
Once you boot, you may have the desire to update @world first thing. However, as of the time of this writing, the latest stage3 for arm64 was built in December 2016. A lot of things in the tool chain will be out of date with what is on the current Portage tree. Once you've booted the board and confirmed that you have an internet connection, you'll want to first run emerge --sync to get the absolute latest tree, then run perl-cleaner --all to get all of your Perl packages up to date.

Network Time Sync
This board does not have a hardware real time clock on board. There are vendors online where you can order RTC modules made for the standard interfaces, but if you don't plan to run one, I highly recommend installing a NTP client.

First, set the initial time using the 'date' command. Date and time will be entered in mmddhhmmyyyy format and the time is in 24-hr format -

As an example, if the time is 10:05PM on 7/31/2017 -

As with most things Gentoo, the NTP daemon is just an emerge away -

Remove the hardware clock service hwclock from the boot runlevel and replace it with the software clock service swclock -

Make sure you have the correct time zone set to the area which most closely matches your locale in /usr/share/zoneinfo -

As an example, if you live in California, you would do -

Install your timezone libraries -

Start the NTP client and add it to the default runlevel -

Where to Get Help
On Internet Relay Chat irc.freenode.net#gentoo-arm irc.freenode.net#gentoo-embedded

On the Gentoo Forums, start a new topic in the Gentoo on ARM forum.

Acknowledgements
Everyone contributing to the arm64 software base.

Especially everyone...