StarFive VisionFive 2

From Gentoo Wiki
Jump to:navigation Jump to:search

This page describes several methods for installing Gentoo on the StarFive VisionFive 2.

Introduction

This page aims to provide a comprehensive introduction to the various methods of installing Gentoo onto Embedded hardware using the VisionFive 2 as an example device.

The majority of the instructions here are not specific to the VisionFive 2 and may be applied to other devices with similar hardware. Please apply some common sense when adapting these instructions for other devices and do not blindly copy and paste commands without understanding what they do.

It should also be noted that the processes described hereafter are not always the most efficient in terms of commands used, and multiple commands may be used with, for example, environment variables that would typically be exported, or repeated additional options where a wrapper would be useful. This is intentional as it is intended to be a learning experience for the reader.

This article is intended to supplement the Embedded Handbook.

Hardware

For a detailed overview of the StarFive hardware, please visit StarFive_VisionFive_2/Hardware

Boot sequence

Important
Do not trust the "ON" label on the boot selector DIP switch as that latter may be soldered in the reverse position as it is the case in the photo here-after. The true states vs switch positions are (with GPIO header seen upper-left):
  • OFF (low or 0 level): switch positioned towards to the "ON" label (switch is on the right)
  • ON (high or 1 level): switch positioned backwards to the "ON" label (switch is on the left)


The boot sequence of a typical RISC-V device is as follows:

  1. When the SoC is powered on, the CPU fetches instructions beginning at address 0x0, where is the BootROM (BROM AKA Primary Program Loader) is located.
  2. The BROM loads the Secondary Program Loader (SPL) from some form of Non Volatile Memory (NVM).
  3. The SPL loads a U-Boot Flattened Image Tree (FIT) image (presumably from the same device). This FIT image contains the U-Boot binary, the Device Tree Blob (DTB) and the OpenSBI binary. This image may be combined with the SPL.
  4. U-Boot loads the Linux Kernel.

In the case of the VisionFive 2/JH7110, the BROM is located on 32k of onboard (on the SoC) memory which may be seen on the SoC block diagram. The BROM is what it is: a ROM (Read Only Memory) so it is impossible to update it.

The VisionFive 2 BROM uses the state of the RGPIO pins on the board to determine which attached storage device (NVM) to load the the U-Boot Secondary Program Loader (U-Boot SPL) (u-boot-spl.bin.normal.out) from. By default this is a partition with a GUID type code of 2E54B353-1271-4842-806F-E436D6AF6985. The number of this partition is irrelevant, though it is typically partition 1.

In its default configuration, the U-Boot SPL then loads the U-Boot FIT image (u-boot.itb) from partition 2 (CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_PARTITION=0x2). When formatting this partition the recommended GUID type code is BC13C2FF-59E6-4262-A352-B275FD6F7172.

The FIT image (u-boot.itb) is a combination of OpenSBI's fw_dynamic.bin, u-boot-nodtb.bin and the device tree blob (jh7110-starfive-visionfive-2-v1.3b.dtb or jh7110-starfive-visionfive-2-v1.2a.dtb).

The VisionFive 2 has a two-switch RGPIO header on the board to select the device to load the firmware from. The configurations are as follows:

Option RGPIO_0 RGPIO_1
Onboard QSPI Flash (factory setting) 0 0
SDIO3.0 (TF / microSD card) 1 0
eMMC 0 1
UART 1 1

First start

Serial connection

Note
The only way to interact with the board is to use a serial link as its HDMI output is not functional (yet). The following is valid for board revision 1.3B.
Important
Notice the RGPIO switches positions as the DIP switch has been mounted upside-down: both RGPIO switches are set in their right position (OFF) thus selecting the onboard SPI FLash as a boot device despite being pushed towards the ON label.

The StarFive VisionFive 2 supports a serial connection (3.3V) through its GPIO pins. Its UART is accessible via the pins 6 (GND), 8 (TX) and 10 (RX) so a crossed-over cable is required:

StarFive VisionFive 2 (1.3B) GPIO pin Serial adapter pin
6 (Ground) Ground
8 (TX) RX
10 (RX) TX

Illustration for a 1.3B board revision with a connected USB adapter (from left to right its pins are GND - black wire, RXD - blue wire, TXD - violet wire, 3.3V + VCC - yellow jumper, 5V - not connected):

Connecting the StarFive VisionFive 2 (rev. 1.3B) to an USB serial adapter

Install a communication program that can handle a serial port connection and transfer files over X-Modem or Y-Modem protocols like net-dialup/minicom (will be used for the rest of the explanations):

root #emerge --ask net-dialup/minicom

Then launch Minicom by specifying the serial device to use (likely /dev/ttyUSB0):

user $minicom -D /dev/ttyUSB0 -b 115200

The serial port connection parameters should be set as being:

  • Speed: 115200 bps
  • Data: 8 bits
  • Parity: None
  • Stop bits: 1 bit
  • No control flow (hardware and software)
Tip
To set those parameters directly in Minicom rather than on the command line, use the keys Ctrl+A then press Z then press O when the menu shows up.

Powering up

As the board's QSPI flash already contains a firmware (preloaded at factory), it is possible to use the board when unboxing it. QSPI flash is too small to contain a full-fledged OS nevertheless it is more than enough to contain an OpenSBI/U-Boot environment (comparable to OpenBOOT for SPARC machines users).

With the serial connection setup in Minicom and the serial cable connected to the StarFive VisionFive 2 UART pins, powering the board up should lead to the following display in Minicom:

U-Boot SPL 2021.10 (Sep 28 2024 - 01:14:56 +0800) 
LPDDR4: 8G version: g8ad50857.
Trying to boot from SPI

OpenSBI v1.2
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|___/_____|
        | |
        |_|

Platform Name             : StarFive VisionFive V2
Platform Features         : medeleg
Platform HART Count       : 5
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 4000000Hz
Platform Console Device   : uart8250
Platform HSM Device       : ---
Platform PMU Device       : ---
Platform Reboot Device    : pm-reset
Platform Shutdown Device  : pm-reset
Platform Suspend Device   : ---
Firmware Base             : 0x40000000
Firmware Size             : 392 KB
Firmware RW Offset        : 0x40000
Runtime SBI Version       : 1.0

Domain0 Name              : root
Domain0 Boot HART         : 1
Domain0 HARTs             : 0*,1*,2*,3*,4*
Domain0 Region00          : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: ()
Domain0 Region01          : 0x0000000040000000-0x000000004003ffff M: (R,X) S/U: ()
Domain0 Region02          : 0x0000000040040000-0x000000004007ffff M: (R,W) S/U: ()
Domain0 Region03          : 0x0000000000000000-0xffffffffffffffff M: (R,W,X) S/U: (R,W,X)
Domain0 Next Address      : 0x0000000040200000
Domain0 Next Arg1         : 0x0000000042200000
Domain0 Next Mode         : S-mode
Domain0 SysReset          : yes
Domain0 SysSuspend        : yes

Boot HART ID              : 1
Boot HART Domain          : root
Boot HART Priv Version    : v1.11
Boot HART Base ISA        : rv64imafdcbx
Boot HART ISA Extensions  : none
Boot HART PMP Count       : 8
Boot HART PMP Granularity : 4096
Boot HART PMP Address Bits: 34
Boot HART MHPM Count      : 2
Boot HART MIDELEG         : 0x0000000000000222
Boot HART MEDELEG         : 0x000000000000b109

U-Boot 2021.10 (Sep 28 2024 - 01:14:56 +0800), Build: jenkins-github_visionfive2_6.6-5

CPU:   rv64imacu_zba_zbb
Model: StarFive VisionFive V2
DRAM:  8 GiB
MMC:   sdio0@16010000: 0, sdio1@16020000: 1
Loading Environment from SPIFlash... SF: Detected gd25lq128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
OK
StarFive EEPROM format v2

--------EEPROM INFO--------
Vendor : StarFive Technology Co., Ltd.
Product full SN: VF7110B1-2318-D008E000-00000000
data version: 0x2
PCB revision: 0xb2
BOM revision: A
Ethernet MAC0 address: 6c:cf:39:00:b0:16
Ethernet MAC1 address: 6c:cf:39:00:b0:17
--------EEPROM INFO--------

In:    serial
Out:   serial
Err:   serial
Model: StarFive VisionFive V2
Net:   eth0: ethernet@16030000, eth1: ethernet@16040000
Hit any key to stop autoboot:  0 
Card did not respond to voltage select! : -110
Card did not respond to voltage select! : -110
starfive_pcie pcie@2B000000: Port link up.
starfive_pcie pcie@2B000000: Starfive PCIe bus probed.
PCI: Failed autoconfig bar 10
starfive_pcie pcie@2C000000: Port link down.
starfive_pcie pcie@2C000000: Starfive PCIe bus probed.
PCI: Failed autoconfig bar 10

Device 0: unknown device

Device 0: unknown device
Tring booting distro ...
Card did not respond to voltage select! : -110
Card did not respond to voltage select! : -110

Device 0: unknown device

Device 0: unknown device
StarFive # 

So far so good, an U-Boot command prompt beginning with StarFive # should show up at the end of the booting process. No worries for error messages like Card did not respond to voltage select! : -110 and Device 0: unknown device, they are normal at this point. Enter something like help to get a list of what valid commands are under the U-Boot shell as a test:

StarFive #help
?         - alias for 'help'
base      - print or set address offset
bdinfo    - print Board Info structure
blkcache  - block cache diagnostics and control
boot      - boot default, i.e., run 'bootcmd'
bootd     - boot default, i.e., run 'bootcmd'
bootefi   - Boots an EFI payload from memory
bootelf   - Boot from an ELF image in memory
booti     - boot Linux kernel 'Image' format from memory
bootm     - boot application image from memory
bootp     - boot image via network using BOOTP/TFTP protocol
bootvx    - Boot vxWorks from an ELF image
cmp       - memory compare
config    - print .config
coninfo   - print console devices and information
cp        - memory copy
cpu       - display information about CPUs
crc32     - checksum calculation
dhcp      - boot image via network using DHCP/TFTP protocol
dm        - Driver model low level access
echo      - echo args to console
editenv   - edit environment variable
eeprom    - EEPROM sub-system
efidebug  - Configure UEFI environment
env       - environment handling commands
erase     - erase FLASH memory
eraseenv  - erase environment variables from persistent storage
exit      - exit script
ext2load  - load binary file from a Ext2 filesystem
(...)
showvar   - print local hushshell variables
size      - determine a file's size
sleep     - delay execution for some time
source    - run script from memory
sysboot   - command to get and boot from syslinux files
test      - minimal test like /bin/sh
tftpboot  - boot image via network using TFTP protocol
tftpput   - TFTP put command, for uploading files to a server
true      - do nothing, successfully
unlz4     - lz4 uncompress a memory region
unzip     - unzip a memory region
usb       - USB sub-system
usbboot   - boot from USB device
version   - print monitor, compiler and linker version

Everything is in good working order at this point. The StarFive VisionFive 2 can be powered down by removing its USB-C power cable (safe operation, will not corrupt anything).

First aid kit

Nothing being perfect, let's take precautions. There is a recovery image available on GitHub that can be used if, for a reason or another, the onboard SPI Flash memory would become corrupt. If that would happen, the recovery image will enable you to start your StarFive VisionFive 2 and reflash it.

First things first, download the most recent recovery binary file on https://github.com/starfive-tech/Tools/tree/master/ recovery (that binary file is named jh7110-recovery-YYYYMMDD.bin, e.g. jh7110-recovery-20230322.bin).

With the StarFive VisionFive 2 board powered down, change the boot selector setting to make the board boot via its UART (i.e. serial link) rather than its SPI Flash memory (RGPIO_0 and RGPIO_1 must be both set to ON, i.e on their left position, GPIO header seen upper-left). Once the boot selector setting has been changed, power the board up again. At this point, Minicom should show a series of "C", meaning that the board is patiently waiting for its firmware image to be sent to it by Minicom via the X-Modem protocol:

U-Boot SPL 2021.10 (Sep 28 2024 - 01:14:56 +0800) 
(C)StarFive                                                                                                                                                                                                                          
CCCCCCCCCCCCCCCCCCCCC 

Within Minicom, transfer the recovery binary file just downloaded from Github using the said X-Modem protocol (Ctrl+A then press S then select xmodem). Once the recovery image has been transferred, the StarFive VisionFive 2 board will automatically boot on it and will present a menu from where an operation to perform can be chosen:

U-Boot SPL 2021.10 (Sep 28 2024 - 01:14:56 +0800) 
(C)StarFive                                                                                                                                                                                                                          
CCCCCCCCCCCCCCCCCCCCC                                                                                                                                                                                                                
JH7110 secondboot version: 230322-c514da9                                                                                                                                                                                            
CPU freq: 1250MHz                                                                                                                                                                                                                    
idcode: 0x1860C8                                                                                                                                                                                                                     
ddr 0x00000000, 4M test                                                                                                                                                                                                              
ddr 0x00400000, 8M test                                                                                                                                                                                                              
DDR clk 2133M, size 8GB                                                                                                                                                                                                              
                                                                                                                                                                                                                                     
*********************************************************                                                                                                                                                                            
****************** JH7110 program tool ******************                                                                                                                                                                            
*********************************************************                                                                                                                                                                            
0: update 2ndboot/SPL in flash                                                                                                                                                                                                       
1: update 2ndboot/SPL in emmc                                                                                                                                                                                                        
2: update fw_verif/uboot in flash                                                                                                                                                                                                    
3: update fw_verif/uboot in emmc                                                                                                                                                                                                     
4: update otp, caution!!!!                                                                                                                                                                                                           
5: exit                                                                                                                                                                                                                              
NOTE: current xmodem receive buff = 0x40000000, 'load 0x********' to change.                                                                                                                                                         
select the function to test: 
Warning
Do not use option 4 (OTP) unless being in good knowledge: an OTP is a One Time Programmable memory or WORM (Write-Once Read Many memory). Once an OTP memory has been programmed ("fused"), it cannot be programmed again unless using specialized equipment or procedures. OTP fuses dot not have a public documentation, see the following discussion thread: https://github.com/starfive-tech/Tools/issues/8

Options 0 and 2 can be used to reprogram the onboard SPI Flash memory to make it bootable again. Options 1 and 2 do exactly the same if you have plugged an eMMC memory module in (at the back of the board).

Updating the stock firmware preloaded at factory

Before attempting to install Gentoo onto the VisionFive 2 it is essential to update the firmware. Depending on the age of the firmware the current (modern) official images from StarFive may fail to load.

There are several methods for updating the firmware, including:

  1. Using the U-Boot console to retrieve and install firmware images via TFTP
  2. Invoking flashcp command (from sys-fs/mtd-utils) on a running system
  3. Sending firmware binaries over a UART console interface
  4. Booting updated firmware off removable media

Path #3 will be taken here as it involves the least additional software and does not require a bootable board.

Important
Updating U-Boot will likely reset the contents of U-Boot variables to their default value. If some of them were customized to have a fully working system it is highly advised to keep a trace of their current value before proceeding!

First, gather the following firmware files from the latest VisionFive2 software release:

Firmware file Base address in SPI flash memory
u-boot-spl.bin.normal.out 0x0
visionfive2_fw_payload.img 0x100000

Second, calculate their CRC-32 with /usr/bin/crc32 (provided by dev-perl/Archive-Zip) on the host, results might defer fomr what is show here:

user $crc32 u-boot-spl.bin.normal.out visionfive2_fw_payload.img
4f92e463        u-boot-spl.bin.normal.out
6949ec27        visionfive2_fw_payload.img

Third, ensure the StarFive VisionFive 2 is set up to boot off the onboard SPI Flash memory: RGPIO_0 and RGPIO_1 switches must, both, be set to their OFF position (i.e. on the right, having the GPIO header seen upper-left). Power the board on and the U-Boot command prompt should be reached after a few seconds. From that prompt there, run the following command to initialize a file transfer via the Y-Modem protocol (calculates a CRC for each chunk of transferred data to ensure no corruption has occurred):

StarFive #loady
## Ready for binary (ymodem) download to 0x60000000 at 115200 bps...
CCCCC

At this point, use Ctrl+A then S in Minicom to enter the file transfer menu, then select Ymodem, then a single firmware file of your choice (beginning with u-boot-spl.bin.normal.out or visionfive2_fw_payload.img is not important as long as both are processed) and wait for the transfer to complete.

StarFive #loady
## Ready for binary (ymodem) download to 0x60000000 at 115200 bps...                                                                                                                                                               
CC0(STX)/0(CAN) packets, 6 retries
## Total Size      = 0x00025a30 = 154160 Bytes

At this point the firmware payload lies in board's RAM but remains to be written on the SPI flash memory. Before committing it to the SPI flash memory, check the firmware payload's CRC-32 as it is seen on the StarFive VisionFive 2 board (the result should match what has been computed on the host):

StarFive #crc32 $loadaddr $filesize
crc32 for 60000000 ... 60025a2f ==> 4f92e463
Tip
$filesize is a shell variable automatically initialized when the transfer completes. A couple of shell variables like $loadaddr are preset at system startup. To see all defined variables use the shell command: printenv.

If the CRC-32 value calculated on the board matches the one calculated on the host, the firmware file contents can be comitted to the onboard SPI flash memory. Depending on which firmware file has been transferred, the command to use is:

  • For u-boot-spl.bin.normal.out:
StarFive #sf update $loadaddr 0x0 $filesize
  • For visionfive2_fw_payload.img:
StarFive #sf update $loadaddr 0x100000 $filesize

Repeat the operations above for the second firmware file. Once both firmware files have been transferred and committed to the onboard SPI flash memory, powercycle the board and the U-Boot command prompt should show up again after a couple of seconds. In the case some badluck would have happened making the board unable to boot on the onboard SPI flash memory, it is possible to recover from the situation using recovery boot image, see section First aid kit.

Prerequisites

The process to build a Gentoo stage 4 for the StarFive VisionFive 2 involve several "big" steps:

  1. Setting up QEMU to make it handle RISC-V binaries
  2. Generating a cross-build envoironment used to generate a Catalyst seed
  3. Using Catalyst to build a fully-functional RISC-V rootfs
  4. Chrooting into the RISC-V rootfs
  5. Customizing the RISC-V rootfs (here is where QEMU userland emulation appears in the landscape)

Configure QEMU-user and binfmt

Tip
To get the benefit of the QEMU/KVM speed-up, add the user (e.g. larry) under which target environment will be built to the kvm group: gpasswd -a larry kvm

QEMU should be configured to, at least, support:

  • full-system emulation (i.e. QEMU_SOFTMMU_TARGETS): AMD64 (x86_64) and RISC-V (risc-v)
  • userland emulation (i.e. QEMU_USER_TARGETS): RISC-V (risc-v)

The very first step is to check what lies inside /etc/portage/make.conf and add the required targets discussed herebefore if they were missing:

FILE /etc/portage/make.conf
QEMU_SOFTMMU_TARGETS="riscv64 x86_64"
QEMU_USER_TARGETS="riscv64"

The next step is to ensure that app-emulation/qemu has, at least, the USE flag static-user enabled as anything outside the chroot will be inaccessible (QEMU binaries should be fully autonomous):

FILE /etc/portage/package.use/qemu
app-emulation/qemu static-user

Now, re-emerge app-emulation/qemu and package it to be able to be re-deployed inside the chrooted target environment:

root #emerge --ask app-emulation/qemu
root #quickpkg app-emulation/qemu

It is time to register the ELF/RISC-V signature to the miscellaneous binary formats known by the Linux kernel to make the kernel invoke the adequate QEMU userland emulator when needed:

root #mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
root #echo ':riscv64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-riscv64:' > /etc/binfmt.d/qemu-riscv64-static.conf

Then restart binfmt support:

  • OpenRC:
    root #rc-service binfmt restart
  • Systemd:
    root #systemctl restart systemd-binfmt

At this point a pseudo-file called riscv64 should appear under /proc/sys/fs/binfmt_misc :

user $ls -l /proc/sys/fs/binfmt_misc
total 0
--w------- 1 root root 0 Nov 12 10:13 register
-rw-r--r-- 1 root root 0 Nov 12 10:13 riscv64
-rw-r--r-- 1 root root 0 Nov  6 10:57 status

The pseudo-file /proc/sys/fs/binfmt_misc/riscv64 contains: Template:CmdRoot

Generate a Cross Build Environment (crossdev)

Rather than using the (already outdated) version of GCC specified in the VisionFive 2 SDK crossdev may instead be used to to build an up-to-date cross-compiler from the Gentoo repository. This will then be used to bootstrap a Catalyst stage, build the kernel, and any required firmware binaries.

Install the sys-devel/crossdev package and generate a RISC-V cross toolchain (see Cross Build Environment for further information):

root #emerge --ask sys-devel/crossdev

Create an ebuild repository for crossdev, preventing it from choosing a (seemingly) random repository to store its packages:

root #mkdir -p /var/db/repos/crossdev/{profiles,metadata}
root #echo 'crossdev' > /var/db/repos/crossdev/profiles/repo_name
root #echo 'masters = gentoo' > /var/db/repos/crossdev/metadata/layout.conf
root #chown -R portage:portage /var/db/repos/crossdev

If the Gentoo ebuild repository is synchronized using Git, or any other method with Manifest files that do not include checksums for ebuilds:

FILE /var/db/repos/crossdev/metadata/layout.conf
masters = gentoo
thin-manifests = true

Instruct Portage and crossdev to use this ebuild repository:

FILE /etc/portage/repos.conf/crossdev.conf
[crossdev]
location = /var/db/repos/crossdev
priority = 10
masters = gentoo
auto-sync = no

Then build the cross-build toolchain:

root #crossdev --target riscv64-unknown-linux-gnu

Once crossdev has built the cross-toolchain, that latter will be installed to /usr/<target> (i.e. /usr/ in the present case). The cross-compiler may be used by prefixing the target to the command, e.g. riscv64-unknown-linux-gnu-gcc:

root #riscv64-unknown-linux-gnu-gcc -v
Using built-in specs.
COLLECT_GCC=riscv64-unknown-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/riscv64-unknown-linux-gnu/14/lto-wrapper
Target: riscv64-unknown-linux-gnu
Configured with: /var/tmp/portage/cross-riscv64-unknown-linux-gnu/gcc-14.2.1_p20241026/work/gcc-14-20241026/configure --host=x86_64-pc-linux-gnu --target=riscv64-unknown-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/riscv64-unknown-linux-gnu/gcc-bin/14 --includedir=/usr/lib/gcc/riscv64-unknown-linux-gnu/14/include --datadir=/usr/share/gcc-data/riscv64-unknown-linux-gnu/14 --mandir=/usr/share/gcc-data/riscv64-unknown-linux-gnu/14/man --infodir=/usr/share/gcc-data/riscv64-unknown-linux-gnu/14/info --with-gxx-include-dir=/usr/lib/gcc/riscv64-unknown-linux-gnu/14/include/g++-v14 --disable-silent-rules --disable-dependency-tracking --with-python-dir=/share/gcc-data/riscv64-unknown-linux-gnu/14/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --disable-libunwind-exceptions --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 14.2.1_p20241026 p3' --with-gcc-major-version-only --enable-libstdcxx-time --enable-lto --disable-libstdcxx-pch --enable-poison-system-directories --with-sysroot=/usr/riscv64-unknown-linux-gnu --disable-gcov --disable-bootstrap --enable-__cxa_atexit --enable-clocale=gnu --disable-multilib --disable-fixed-point --with-abi=lp64d --enable-libgomp --disable-libssp --disable-libada --disable-systemtap --disable-valgrind-annotations --disable-vtable-verify --disable-libvtv --with-zstd --without-isl --disable-libsanitizer --enable-default-pie --enable-host-pie --disable-host-bind-now --enable-default-ssp --disable-fixincludes
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.2.1 20241026 (Gentoo 14.2.1_p20241026 p3)

Building a system image

There are two philosophies when it comes to installing an operating system onto a SBC / embedded device such as the VisionFive 2.

The first involves writing a static system image, typically a squashfs, onto some form of media (typically eMMC, though NVMe is on the rise). The initramfs is able to load this image into RAM and use it as a rootfs; when an update is required the whole image is replaced as a single operation. There are advantages to this approach, particularly for embedded devices where users are not expected to update individual packages and recovery 'in-the-field' may be impractical, or to provide an A/B partition layout for updates. Systems configured in this way are also resilient when it comes to unexpected shutdowns as the only time that the rootfs storage volume is performing writes is when this image is being updated. This approach will be described as an 'embedded' installation going forward and called out where possible.

The second approach involves writing a rootfs onto some accessible storage media the device and using a package manager to install and update packages as required. For a Gentoo system this approach is more flexible and allows for a more traditional Linux experience, but requires more effort to set up and maintain. This approach will be described as a 'traditional' installation going forward and called out where possible.

The process of generating a system image for a Gentoo installation on the VisionFive 2 may be broadly described as follows:

  • Gather the installation files and generate a cross-compiler
    • Check out the VisionFive 2 SDK OR kernel sources (until the upstream kernel has full hardware support)
    • Build a cross-compiler and use it to build a kernel, initramfs, and FIT image
  • Generate a Gentoo rootfs
    • Unpack and customize a Gentoo Stage 3 tarball to create a Gentoo rootfs
    • Use Catalyst to generate an image from scratch
  • Load the Flattened Image Tree (FIT) onto the device
  • Write a Gentoo rootfs to the selected storage medium.

Generate a cross-compiler

A cross-compiler will typically also be required as the VisionFive 2 is a RISC-V device and most users will be running an x86_64 (amd64) host.

Check out the VisionFive 2 BSP

Tip
See https://rvspace.org/en/project/JH7110_Upstream_Plan to get the upstreaming process status.

The VisionFive 2 Board Support Package (BSP) (sometimes called an SDK by manufacturers) is a git repository containing a collection of scripts (and git submodules) that may be used to bootstrap a cross-compiler and build a Linux kernel, initramfs, rootfs, U-Boot, and FIT image for the VisionFive 2.

As most consumers of this article already use Gentoo, this step is not essential; Gentoo users have the option of using crossdev (from sys-devel/crossdev) to get an up-to-date riscv64 cross-compiler with which to build the kernel, initramfs, and FIT image. However, the SDK is still useful: it provides a convenient way to build U-Boot and a rootfs, contains a great deal of information about how the developers intend for images to be written to the device, and, for inexperienced embedded users, provides a way to generate a guaranteed bootable image and a less-complex approach which be preferable.

The VisionFive 2 BSP repository is available at: https://github.com/starfive-tech/VisionFive2

Use the following commands to check out the repository:

Cloning into 'VisionFive2'...
remote: Enumerating objects: 4479, done.
remote: Counting objects: 100% (603/603), done.
remote: Compressing objects: 100% (311/311), done.
remote: Total 4479 (delta 331), reused 552 (delta 292), pack-reused 3876
Receiving objects: 100% (4479/4479), 290.58 MiB | 6.31 MiB/s, done.
Resolving deltas: 100% (2457/2457), done.
user $cd VisionFive2
user $git submodule update --init --recursive
Submodule 'buildroot' (https://github.com/starfive-tech/buildroot.git) registered for path 'buildroot'
Submodule 'linux' (https://github.com/starfive-tech/linux.git) registered for path 'linux'
Submodule 'opensbi' (https://github.com/starfive-tech/opensbi.git) registered for path 'opensbi'
Submodule 'soft_3rdpart' (https://github.com/starfive-tech/soft_3rdpart.git) registered for path 'soft_3rdpart'
Submodule 'u-boot' (https://github.com/starfive-tech/u-boot.git) registered for path 'u-boot'
Cloning into '/data/development/visionfive/VisionFive2/buildroot'...
Cloning into '/data/development/visionfive/VisionFive2/linux'...
Cloning into '/data/development/visionfive/VisionFive2/opensbi'...
Cloning into '/data/development/visionfive/VisionFive2/soft_3rdpart'...
Cloning into '/data/development/visionfive/VisionFive2/u-boot'...
Submodule path 'buildroot': checked out '762ee9bc4e1fbdaf09675acaed9516d6c136d5b1'
Submodule path 'linux': checked out 'a87c6861c6d96621026ee53b94f081a1a00a4cc7'
Submodule path 'opensbi': checked out 'c6a092cd80112529cb2e92e180767ff5341b22a3'
Submodule path 'soft_3rdpart': checked out 'cd7b50cd9f9eca66c23ebd19f06a172ce0be591f'
Submodule path 'u-boot': checked out '688befadf1d337dee3593e6cc0fe1c737cc150bd'

Use the BSP to build everything

For inexperienced embedded developers it may be desirable to use the VisionFive 2 SDK to build the kernel, initramfs, U-Boot, etc. This is not the recommended approach as the BSP uses outdated dependencies that include a number of bugs and issues that have been fixed in the upstream components.

Use the SDK's build scripts to build a cross-compiler, kernel, initramfs, U-Boot, etc.

user $make -j$(nproc)

The output of this process (kernel, initramfs, device tree blobs [dtb], OR the FIT image containing them) may be used to boot a Gentoo rootfs; if this is all that is desired there is no need to build custom Gentoo versions.

Use crossdev

Rather than using the (already outdated) version of GCC specified in the VisionFive 2 SDK crossdev may instead be used to to build an up-to-date cross-compiler from the Gentoo repository. This will then be used to bootstrap a Catalyst stage, build the kernel, and any required firmware binaries.

Install the sys-devel/crossdev package and generate a RISC-V cross toolchain (see Cross Build Environment for further information):

root #emerge --ask sys-devel/crossdev

Create an ebuild repository for crossdev, preventing it from choosing a (seemingly) random repository to store its packages:

root #mkdir -p /var/db/repos/crossdev/{profiles,metadata}
root #echo 'crossdev' > /var/db/repos/crossdev/profiles/repo_name
root #echo 'masters = gentoo' > /var/db/repos/crossdev/metadata/layout.conf
root #chown -R portage:portage /var/db/repos/crossdev

If the Gentoo ebuild repository is synchronized using Git, or any other method with Manifest files that do not include checksums for ebuilds:

FILE /var/db/repos/crossdev/metadata/layout.conf
masters = gentoo
thin-manifests = true

Instruct Portage and crossdev to use this ebuild repository:

FILE /etc/portage/repos.conf/crossdev.conf
[crossdev]
location = /var/db/repos/crossdev
priority = 10
masters = gentoo
auto-sync = no
root #crossdev --target riscv64-unknown-linux-gnu

Once crossdev has built the cross-toolchain it will be installed to /usr/<target>. The cross-compiler may be used by prefixing the target to the command, e.g.

user $riscv64-unknown-linux-gnu-gcc

Until the VisionFive 2 is fully supported by the upstream Linux Kernel it is necessary to use the VisionFive 2 kernel fork. Clone the repository:

user $cd linux
user $git checkout tags/VF2_v2.11.5
Build the kernel

In order for the VisionFive 2 to run Linux a Linux Kernel is required. This is built using the cross-compiler provided by crossdev.

Note
It's recommended to use GCC 12 to build the VisionFive 2's 5.15.0 kernel
Note
CONFIG_SECCOMP=y is not set in the defconfig and should be manually enabled if packages such as net-libs/webkit-gtk will be installed with USE=seccomp.
Note
There are some out-of-tree kernel modules (jpu/venc/vdec) that should be installed in addition to this kernel. These modules are available from the VisionFive 2 soft_3rdpart repo, or by emerging media-video/vf2vpudev from the bingch overlay
user $cd linux
user $HWBOARD_FLAG=HWBOARD_VISIONFIVE2 ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- make starfive_visionfive2_defconfig
user $HWBOARD_FLAG=HWBOARD_VISIONFIVE2 ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- make menuconfig
user $HWBOARD_FLAG=HWBOARD_VISIONFIVE2 ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- make -j$(nproc) vmlinux all modules

On such error: use of enum ‘gsi_iterator_update’ without previous declaration

Just use this patch: https://lore.kernel.org/lkml/DB6P189MB05681E9F4785DF2758B9875B9CA49@DB6P189MB0568.EURP189.PROD.OUTLOOK.COM/t/

Skipping ahead a bit, once a rootfs is available, install the kernel modules to the rootfs:

root #HWBOARD_FLAG=HWBOARD_VISIONFIVE2 ARCH=riscv INSTALL_MOD_PATH=/path/to/rootfs make modules_install

Once a Gentoo initramfs has been generated, mkimage from dev-embedded/u-boot-tools may be used to consolidate the kernel, initramfs, and dtb into a single file that U-Boot can load. This is not required; the kernel, initramfs, and dtb may also be loaded separately by U-Boot.

root #echo "sys-apps/dtc yaml" > /etc/portage/package.use/device-tree-compiler
root #emerge --ask sys-apps/dtc dev-embedded/u-boot-tools
Note
Update the paths of the kernel, initramfs, and dtb in the .its file before running mkimage.
user $mkimage -f visionfive2-fit-image.its -A riscv -O linux -T flat_dt gentoo.fit

Root filesystem

There are several methods that may be used to create or obtain a Gentoo root filesystem for the JH7110 SoC. The simplest method is simply downloading and unpacking a stage3 tarball from https://www.gentoo.org/downloads/, if a suitable one is available:

root #mkdir rootfs
root #tar xpvf stage3-*.tar.xz --xattrs-include='*.*' --numeric-owner -C rootfs

This example will assume that this has not been selected and instead Catalyst will be used to generate an appropriate stage3 tarball from scratch. If using an upstream stage3 tarball is desired, skip ahead to customising the rootfs.

It is possible directly use the crossdev root under /usr to build a rootfs; it is broadly similar to using Catalyst however instead of generating a seed tarball the rootfs is built by chrooting directly into the crossdev root. There are some advantages to this approach, particularly that it is possible to save a significant amount of time.

While this may be faster for quick development or building for a single device, if intending to target multiple configurations it is usually better to use Catalyst as a generic stage 1 image may be created and used as the base for multiple stage 3 images. Using Catalyst also provides better isolation between the host and target systems.

Build a seed tarball

To create a stage3 tarball, Calalyst requires a seed tarball. Catalyst will chroot into the seed and emerge packages for the new stage to ensure that packages generated for stage tarballs are isolated from the host system.

This example will build a seed tarball from scratch; an appropriate stage3 tarball from upstream may be placed in /var/tmp/catalyst/builds/default and used instead.

Set the system profile

root #PORTAGE_CONFIGROOT=/usr/riscv64-unknown-linux-gnu eselect profile list
Available profile symlink targets:
  [1]   default/linux/riscv/20.0/rv64gc/lp64d (stable)
  [2]   default/linux/riscv/20.0/rv64gc/lp64d/desktop (dev)
  [3]   default/linux/riscv/20.0/rv64gc/lp64d/desktop/gnome (dev)
  [4]   default/linux/riscv/20.0/rv64gc/lp64d/desktop/gnome/systemd (dev)
  [5]   default/linux/riscv/20.0/rv64gc/lp64d/desktop/gnome/systemd/merged-usr (dev)
  [6]   default/linux/riscv/20.0/rv64gc/lp64d/desktop/plasma (dev)
  [7]   default/linux/riscv/20.0/rv64gc/lp64d/desktop/plasma/systemd (dev)
  [8]   default/linux/riscv/20.0/rv64gc/lp64d/desktop/plasma/systemd/merged-usr (dev)
  [9]   default/linux/riscv/20.0/rv64gc/lp64d/desktop/systemd (dev)
  [10]  default/linux/riscv/20.0/rv64gc/lp64d/desktop/systemd/merged-usr (dev)
  [11]  default/linux/riscv/20.0/rv64gc/lp64d/systemd (stable)
  [12]  default/linux/riscv/20.0/rv64gc/lp64d/systemd/merged-usr (stable)
  [13]  default/linux/riscv/20.0/rv64gc/lp64 (stable)
  [14]  default/linux/riscv/20.0/rv64gc/lp64/desktop (dev)
  [15]  default/linux/riscv/20.0/rv64gc/lp64/desktop/gnome (dev)
  [16]  default/linux/riscv/20.0/rv64gc/lp64/desktop/gnome/systemd (dev)
  [17]  default/linux/riscv/20.0/rv64gc/lp64/desktop/gnome/systemd/merged-usr (dev)
  [18]  default/linux/riscv/20.0/rv64gc/lp64/desktop/plasma (dev)
  [19]  default/linux/riscv/20.0/rv64gc/lp64/desktop/plasma/systemd (dev)
  [20]  default/linux/riscv/20.0/rv64gc/lp64/desktop/plasma/systemd/merged-usr (dev)
  [21]  default/linux/riscv/20.0/rv64gc/lp64/desktop/systemd (dev)
  [22]  default/linux/riscv/20.0/rv64gc/lp64/desktop/systemd/merged-usr (dev)
  [23]  default/linux/riscv/20.0/rv64gc/lp64/systemd (stable)
  [24]  default/linux/riscv/20.0/rv64gc/lp64/systemd/merged-usr (stable)
  [25]  default/linux/riscv/20.0/rv64gc/multilib (exp)
  [26]  default/linux/riscv/20.0/rv64gc/multilib/systemd (exp)
  [27]  default/linux/riscv/20.0/rv64gc/multilib/systemd/merged-usr (exp)
  [28]  default/linux/riscv/20.0/rv64gc/lp64d/musl (dev)
  [29]  default/linux/riscv/20.0/rv64gc/lp64/musl (dev)
root #PORTAGE_CONFIGROOT=/usr/riscv64-unknown-linux-gnu eselect profile set 10
Warning
As of profile 23.0, the equivalent profile is 35: default/linux/riscv/23.0/rv64/lp64d/systemd (stable) as documented in the 23.0 Profile update table.

If a profile marked experimental (exp) is desired, use the --force flag to enable the profile.

To prevent errors from occurring while building the seed, the following USE flags should be set to prevent conflicts over the default su provider:

root #mkdir /usr/riscv64-unknown-linux-gnu/etc/portage/package.use
root #echo "sys-apps/util-linux -su" > /usr/riscv64-unknown-linux-gnu/etc/portage/package.use/system

or

root #sed -i -e "s:-pam::" /usr/riscv64-unknown-linux-gnu/etc/portage/make.conf

Emerge the system:

root #riscv64-unknown-linux-gnu-emerge -va1 @system --keep-going
Note
At this point in the process, the Catalyst stage generation may be skipped and instead the system may be built by chrooting into the crossdev environment.

Create a seed tarball:

root #cd /usr/riscv64-unknown-linux-gnu/
root #tar -cvJf /tmp/riscv64-glibc-seed.tar.xz *

Catalyst

Install catalyst:

root #emerge --ask dev-util/catalyst
Important
As of July 2023 Catalyst is only supported as a live ebuild release, See the Catalyst article for more information.

Create a Catalyst work directory, move the seed tarball to Catalyst's workdir, and build a Portage snapshot:

root #mkdir -p /var/tmp/catalyst/builds
root #mv /tmp/riscv64-glibc-seed.tar.xz /var/tmp/catalyst/builds/
root #emerge --sync
root #mkdir -p /var/tmp/catalyst/repos; pushd /var/tmp/catalyst/repos/
root #git clone --mirror /var/db/repos/gentoo
root #popd
root #catalyst --snapshot stable
18 May 2023 10:31:46 AEST: NOTICE  : Loading configuration file: /etc/catalyst/catalyst.conf
NOTICE:catalyst:Loading configuration file: /etc/catalyst/catalyst.conf
18 May 2023 10:31:46 AEST: NOTICE  : conf_values[options] = ['autoresume', 'bindist', 'kerncache', 'pkgcache', 'seedcache']
NOTICE:catalyst:conf_values[options] = ['autoresume', 'bindist', 'kerncache', 'pkgcache', 'seedcache']
18 May 2023 10:31:46 AEST: NOTICE  : >>> /usr/bin/git -C /var/tmp/catalyst/repos/gentoo.git fetch --quiet --depth=1
NOTICE:catalyst:>>> /usr/bin/git -C /var/tmp/catalyst/repos/gentoo.git fetch --quiet --depth=1
18 May 2023 10:31:46 AEST: NOTICE  : >>> /usr/bin/git -C /var/tmp/catalyst/repos/gentoo.git update-ref HEAD FETCH_HEAD
NOTICE:catalyst:>>> /usr/bin/git -C /var/tmp/catalyst/repos/gentoo.git update-ref HEAD FETCH_HEAD
18 May 2023 10:31:46 AEST: NOTICE  : >>> /usr/bin/git -C /var/tmp/catalyst/repos/gentoo.git gc --quiet
NOTICE:catalyst:>>> /usr/bin/git -C /var/tmp/catalyst/repos/gentoo.git gc --quiet
18 May 2023 10:31:47 AEST: NOTICE  : Creating gentoo tree snapshot afe106ae95ed7ba6536c870774c1b7e62d940ebd from /var/tmp/catalyst/repos/gentoo.git
NOTICE:catalyst:Creating gentoo tree snapshot afe106ae95ed7ba6536c870774c1b7e62d940ebd from /var/tmp/catalyst/repos/gentoo.git
18 May 2023 10:31:47 AEST: NOTICE  : >>> /usr/bin/git -C /var/tmp/catalyst/repos/gentoo.git archive --format=tar afe106ae95ed7ba6536c870774c1b7e62d940ebd |
NOTICE:catalyst:>>> /usr/bin/git -C /var/tmp/catalyst/repos/gentoo.git archive --format=tar afe106ae95ed7ba6536c870774c1b7e62d940ebd |
18 May 2023 10:31:47 AEST: NOTICE  :     /usr/bin/tar2sqfs /var/tmp/catalyst/snapshots/gentoo-afe106ae95ed7ba6536c870774c1b7e62d940ebd.sqfs -q -f -j1 -c gzip
NOTICE:catalyst:    /usr/bin/tar2sqfs /var/tmp/catalyst/snapshots/gentoo-afe106ae95ed7ba6536c870774c1b7e62d940ebd.sqfs -q -f -j1 -c gzip
18 May 2023 10:31:55 AEST: NOTICE  : Wrote snapshot to /var/tmp/catalyst/snapshots/gentoo-  .sqfs
NOTICE:catalyst:Wrote snapshot to /var/tmp/catalyst/snapshots/gentoo-afe106ae95ed7ba6536c870774c1b7e62d940ebd.sqfs

Create the Catalyst spec files that match the desired stage type:

Replace afe106ae95ed7ba6536c870774c1b7e62d940ebd in snapshot_treeish with the commit id that was given when running catalyst -s stable

root #cd /var/tmp/catalyst
FILE stage1-riscv64-systemd-mergedusr.spec
subarch: rv64_lp64d
target: stage1
version_stamp: systemd-mergedusr-20230518
interpreter: /usr/bin/qemu-riscv64
rel_type: default
profile: default/linux/riscv/20.0/rv64gc/lp64/systemd/merged-usr
snapshot_treeish: afe106ae95ed7ba6536c870774c1b7e62d940ebd
source_subpath: riscv64-glibc-seed
compression_mode: pixz
decompressor_search_order: xz bzip2
update_seed: yes
update_seed_command: -uDN @world
FILE stage3-riscv64-systemd-mergedusr.spec
subarch: rv64_lp64d
target: stage3
version_stamp: systemd-mergedusr-20230518
interpreter: /usr/bin/qemu-riscv64
rel_type: default
profile: default/linux/riscv/20.0/rv64gc/lp64/systemd/merged-usr
snapshot_treeish: afe106ae95ed7ba6536c870774c1b7e62d940ebd
source_subpath: default/stage1-rv64_lp64d_systemd-mergedusr-20230518.tar.xz
compression_mode: pixz
decompressor_search_order: xz bzip2

Finally, using Catalyst, build a Stage 1 image from the seed tarball, and a Stage 3 image from the Stage 1 image:

root #catalyst -f stage1-riscv64-systemd-mergedusr.spec
root #catalyst -f stage3-riscv64-systemd-mergedusr.spec
Tip
If @system fails to build try checking out the releng repo and setting the portage_confdir variable to its location.
root #git clone -o upstream https://github.com/gentoo/releng.git
root #echo "portage_confdir = /path/to/releng/releases/portage/stages-qemu" >> /var/tmp/catalyst/builds/default/stage1-riscv64-musl-openrc.spec

This ensures that the most up-to-date portage configuration is available to the build process.

If this fails, and a suitable stage 3 image is available, try using that as the seed. If that fails, or is unavailable, ask for support in #gentoo-releng.

If a stage successfully builds the output will be located at /var/tmp/catalyst/builds/default.

Customise the RootFS

Once a a stage 3 image has been obtained or constructed the next task is personalisation of the root filesystem. This involves mounting the root filesystem, followed by executing a chroot command to enter it. For the purpose of the following commands, it is assumed that the root filesystem is stored at /var/tmp/catalyst/builds/default/stage3-riscv64-systemd-mergedusr-20230518.tar.xz.

root #mkdir rootfs
root #tar xpvf /var/tmp/catalyst/builds/default/stage3-*.tar.xz --xattrs-include='*.*' --numeric-owner -C rootfs

Emerge QEMU into the target and chroot; customise the system image (at least set the root password and add a regular user). Proceed with a typical Stage 3 configuration outside of Kernel and Bootloader.

root #mount --bind rootfs rootfs
root #cd rootfs
root #ROOT=$PWD/ emerge --usepkgonly --oneshot --nodeps qemu
root #mount --bind /proc proc
root #mount --bind /sys sys
root #mount --bind /dev dev
root #mount --bind /dev/pts dev/pts
root #mkdir -p var/db/repos/gentoo
root #mount --bind /var/db/repos/gentoo var/db/repos/gentoo
root #mkdir -p usr/src/linux
root #mount --bind ../linux usr/src/linux
root #chroot . /bin/bash --login

To enable accelerated graphics, add the bingch ebuild repository to utilise its custom ebuilds for the Imagination BXE-4-32 GPU.

root #eselect repository add bingch git https://gitlab.com/bingch/gentoo-overlay.git
root #emerge --sync
root #echo 'media-libs/mesa::bingch vulkan' > /etc/portage/package.use/powervr_graphics
root #echo 'VIDEO_CARDS="imagination"' >> /etc/portage/make.conf
root #echo 'media-libs/img-gpu-powervr-bin restricted' > /etc/portage/package.license/powervr-gpu-blobs
Note
LLVM 15 is required to emerge bingch's customized mesa, (or USE="-llvm" which results in a slower swrast which is not ideal for X11); until the Imagination BXE-4-32 is well supported upstream it is unlikely that ::gentoo will contain a version of LLVM that can build the older Mesa releases.
root #emerge --ask media-libs/mesa::bingch

Install the kernel and modules to the rootfs image:

root #pushd /usr/src/linux
root #make install && make modules_install

Generate an initramfs from within the chroot using your rootfs and a tool like Dracut:

root #dracut --kver 5.15.0

Finally, exit the chroot and recursively unmount it:

root #exit
root #umount -R rootfs

Bootloader configuration

There are several methods of of configuring the U-Boot bootloader, each with their own advantages and disadvantages.

An external extlinux configuration file (also known as U-Boot Standard Boot) to determine bootup parameters. This enables U-Boot to load a dynamic configuation from the disk, enabling the user to amend the bootloader configuration without requiring changes to the U-Boot environment.

U-boot is capable of reading from ext2, ext4 and FAT filesystems; each of these filesystems may be used to store the kernel and extlinux/syslinux configuration file.

Regardless of the choice of filesystem, each of these files must be located in a directory called /boot on the root of the partition that it is located. This means that the kernel and extlinuxconfiguration file must be located at /boot/Image and /boot/extlinux/extlinux.conf respectively.

Within the unpacked rootfs, with the kernel at /boot/Image, create a file at /boot/extlinux/extlinux.conf that looks similar to this:

FILE /boot/extlinux/extlinux.conf
label default
    linux /Image
    append root=/dev/mmcblk0p2 rootwait console=ttyS0,115200 earlycon=sbi debug

The extlinux configuration file does not support FIT images.

The bootloader may also be configured by saving the U-Boot environment. This method of configuring the bootloader offers the most control, but requires that the user has a good understanding of the U-Boot environment variables and the boot process.

Imaging the device

There are several ways to get an image onto the device with the most straightforward method being simply writing an image to a TF (MicroSD) card. The VisionFive 2 has a built-in TF card reader, so this is a matter of imaging the card, inserting the card, selecting the appropriate boot option via the RGPIO switches, and powering on the device.

An alternate method where an image is loaded via TFTP will be described here.

The following files may required to boot the VF2 board, depending on the selected boot method:

├── visionfive2_fw_payload.img
├── image.fit
├── initramfs.cpio.gz
├── u-boot-spl.bin.normal.out
├── linux/arch/riscv/boot
    ├── dts
    │   └── starfive
    │       ├── jh7110-visionfive-v2-ac108.dtb
    │       ├── jh7110-visionfive-v2.dtb
    │       ├── jh7110-visionfive-v2-wm8960.dtb
    │       ├── vf2-overlay
    │       │   └── vf2-overlay-uart3-i2c.dtbo
    └── Image.gz

A note about firmware

While the process of imaging the SPI NAND was documented earlier in this guide, it is not the only location that firmware may be loaded from. Depending on the configuration of the RGPIO switches (and saved or compiled-in U-Boot configuration) the VF2 device may attempt to load firmware from partitions on a variety of block devices or over UART.

M.2 installation using a single rootfs partition

This example utilises the QSPI NAND to store the firmware and U-Boot environment alongside with a single rootfs partition on the M.2 NVMe SSD, used to store the rootfs with a FIT image being stored (and updated if necessary) under /boot. This configuration is broadly analogous to booting a legacy desktop with a monolithic partition and GRUB or LILO installed to the MBR with some additional, manual, bootloader configuration.

It is not a requirement to store the FIT image on the same block device as the rootfs, however it is often convenient to do so. A potential downside to this approach is that U-Boot only knows how to read ext2, ext4 and FAT file systems which restricts the choice of rootfs. It is entirely possible to make /boot another partition and store the FIT image elsewhere. If this option is selected ensure that partition numbers and device names are updated (or, ideally, use UUIDs for the rootfs in the kernel cmdline).

It is assumed that the user has some familiarity with running a TFTP server such as net-ftp/tftp-hpa.

To begin, validate the produced image.fit. Set environment parameters, load the image from the TFTP sever into memory, and boot it:

StarFive #setenv bootfile vmlinuz; setenv fdtcontroladdr 0xffffffffffffffff; setenv serverip 192.168.1.x; setenv ipaddr 192.168.1.x;
StarFive #tftpboot ${loadaddr} ${serverip}:image.fit;
StarFive #bootm start ${loadaddr};bootm loados ${loadaddr};run chipa_set_linux;run cpu_vol_set;booti 0x40200000 0x46100000:${filesize} 0x46000000
## Loading kernel from FIT Image at a0000000 ...
   Using 'config-1' configuration
   Trying 'vmlinux' kernel subimage
     Description:  vmlinux
     Type:         Kernel Image
     Compression:  uncompressed
     Data Start:   0xa00000c8
     Data Size:    24161280 Bytes = 23 MiB
     Architecture: RISC-V
     OS:           Linux
     Load Address: 0x40200000
     Entry Point:  0x40200000
   Verifying Hash Integrity ... OK
## Loading fdt from FIT Image at a0000000 ...
   Using 'config-1' configuration
   Trying 'fdt' fdt subimage
     Description:  unavailable
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0xa7f57d50
     Data Size:    48366 Bytes = 47.2 KiB
     Architecture: RISC-V
     Load Address: 0x46000000
     Hash algo:    sha256
     Hash value:   dae1bdb73c5a4806cc8ff17df2552c3152c5d858e76f89212a3de4714e63e40b
   Verifying Hash Integrity ... sha256+ OK
   Loading fdt from 0xa7f57d50 to 0x46000000
   Booting using the fdt blob at 0x46000000
## Loading loadables from FIT Image at a0000000 ...
   Trying 'ramdisk' loadables subimage
     Description:  buildroot initramfs
     Type:         RAMDisk Image
     Compression:  uncompressed
     Data Start:   0xa170ad7c
     Data Size:    109367045 Bytes = 104.3 MiB
     Architecture: RISC-V
     OS:           Linux
     Load Address: 0x46100000
     Entry Point:  unavailable
     Hash algo:    sha256
     Hash value:   9b07bf94f17c3ae38607e09944ebd036c962e91d731d2b81231088a7e56ad46e
   Verifying Hash Integrity ... sha256+ OK
   Loading loadables from 0xa170ad7c to 0x46100000
   Loading Kernel Image
## Flattened Device Tree blob at 46000000
   Booting using the fdt blob at 0x46000000
   Using Device Tree in place at 0000000046000000, end 000000004600eced

Starting kernel ...

If the device successfully boots using the generated FIT image it may be used to install the rootfs onto attached storage.

Use mkstage4 to generate a rootfs tarball:

root #pushd rootfs
root #mkstage4 -C gz -t $(pwd) ../work/visionfive2-rootfs
root #popd

Using the initramfs UART console, partition the NVMe device, set an IP, and copy the rootfs tarball to the device:

#gdisk /dev/nvme0n1
#mkdir /mnt/gentoo
#ip addr add 192.168.1.x/24 dev eth1
#ip route add default via 192.168.1.1 dev eth1
#scp larry@buildhost:visionfive2-rootfs.tar.gz /mnt/gentoo/

Unpack the tarball and configure the stage3 as usual. Copy image.fit to /boot. The FIT image produced by the VF2 SDK may be used to boot the Gentoo installation however, if desired, a new FIT image may be generated using the Gentoo kernel and device tree.

#tar -xpf visionfive2-rootfs.tar.gz -C /mnt/gentoo

It is recommended that the dhcpcd (or an alternative DHCP client) and sshd services be enabled at this time.

From a U-Boot console, scan and identify the NVMe device:

StarFive #nvme scan
StarFive #nvme info
Device 0: Vendor: 0x1179 Rev: 10604107 Prod: Y1BFC3Z2F8R3
            Type: Hard Disk
            Capacity: 976762.3 MB = 953.8 GB (2000409264 x 512)

As U-Boot is able to see the device, configure the bootloader to add an NVMe boot target.

Note
It doesn't appear to be possible to load a FIT image using a sysboot/extlinux/standard boot configuration. It would be ideal for this to be configured dynamically rather than requiring a U-Boot configuration change if (e.g.) the kernel cmdline needs to be updated. Investigate this further and add to the wiki!

First, load the image manually from disk:

StarFive #setenv bootfile vmlinuz; setenv fileaddr a0000000; setenv fdtcontroladdr 0xffffffffffffffff; nvme scan; ext4load nvme 0:1 $fileaddr boot/image.fit
StarFive #bootm start ${fileaddr};bootm loados ${fileaddr};run chipa_set_linux;run cpu_vol_set;booti 0x40200000 0x46100000:${filesize} 0x46000000

To debug a failing boot, try a combination of the following additional kernel cmdline options:

  • PID1 fails to start: systemd.log_level=debug systemd.log_target=console
  • Essential service fails to start after control handed to systemd: systemd.journald.forward_to_console=1

The 'debug' option may be removed from the default 'bootargs' environment setting if desired, once the system is booting successfully.

Once safe commands for booting the system are identified they need to be saved to the SPI flash as macros that can be easily run; several variables will be saved to make the boot process easier to parse and debug. It is important to note that any lines containing other macros or environment variables must be enclosed in ' to ensure that they are expanded correctly at runtime. The following commands may be used to save the boot commands to the SPI flash:

StarFive #setenv nvme_bootargs 'setenv bootargs ${bootargs} root=/dev/nvme0n1p1 rw'
StarFive #setenv nvme_fitload 'setenv bootfile vmlinuz; setenv fdtcontroladdr 0xffffffffffffffff; nvme scan; ext4load nvme 0:1 $loadaddr boot/gentoo.fit'
StarFive #setenv nvme_bootfit 'bootm start ${loadaddr};bootm loados ${loadaddr};run chipa_set_linux;run cpu_vol_set;booti 0x40200000 0x46100000:${filesize} 0x46000000'
StarFive #setenv bootcmd_nvme0 'run nvme_bootargs nvme_fitload nvme_bootfit'
StarFive #setenv boot_targets 'mmc0 dhcp nvme0'
StarFive #saveenv
StarFive #reset
Saving Environment to SPI Flash...
Erasing SPI flash...Writing to SPI flash...done

Upon the next boot the device will load the kernel and initramfs from the NVMe device without interaction.

TF Card/eMMC

In this example the BROM is instructed to load firmware from an attached TF/eMMC device. It is very similar to the the way that aarch64 devices are imaged due to the similar boot mechanisms. The TF card will be partitioned using GPT.

An image will be generated manually on-disk which may then be installed onto the TF card using tools such as dd.

Create an image and mount it on an available loop device:

user $fallocate -l 8G visionfive2.img
user $doas losetup --find --show visionfive2.img
/dev/loop0
Tip
The sectors used below are pulled from the VF2 BSP image generation script. The numbers assume a 512 Byte sector size with partitions aligned to 2048-sector boundaries. It is possible to use a different sector size, refer to the earlier guidance around the SoC's boot sequence.
root #
sgdisk --clear --set-alignment=2 \
  --new=1:4096:8191    --change-name=1:"spl"   --typecode=1:2E54B353-1271-4842-806F-E436D6AF6985 \
  --new=2:8192:16383   --change-name=2:"uboot" --typecode=2:5B193300-FC78-40CD-8002-E86C45580B47 \
  --new=3:16385:614399 --change-name=3:"image" --typecode=3:EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 \
  --new=4:614400:0     --change-name=4:"root"  --typecode=4:0FC63DAF-8483-4772-8E79-3D69D8477DE4 \
  /dev/loop0
Creating new GPT entries in memory.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
Note
If the TF card is larger than the size of the disk image (which is recommended; it saves a lot of time when imaging) resize2fs or a similar utility depending on file system may be used to resize the rootfs partition later.

Write the SPL and U-Boot

Note
It is possible to combine the SPL and U-Boot into a single image FIT image. This will require a different partition layout but should otherwise be similar to this example. This is not covered here, but should be. Please help out by updating the wiki!

The SPL and U-Boot may be written to the image using dd.

root #dd if=u-boot/spl/u-boot-spl.bin of=/dev/loop0p1
root #dd if=u-boot/u-boot.itb of=/dev/loop0p2

Format and mount the boot and root partitions

The boot partition may be formatted with mkfs.fat, and the root partition with mkfs.ext4. The following commands may be used to format and mount the partitions:

root #mkfs.fat -F 32 -n boot /dev/loop0p3
root #mkfs.ext4 -L root /dev/loop0p4
Tip
For a more conventional /boot layout, make /boot a symlink to /mnt/sdcard/boot and mount sdcard at /mnt/sdcard.
root #mkdir -p /mnt/visionfive2
root #mount /dev/loop0p4 /mnt/visionfive2
root #cp -a rootfs/* /mnt/visionfive2
root #mkdir -p /mnt/visionfive2/mnt/sdcard
root #mount /dev/loop0p3 /mnt/visionfive2/mnt/sdcard
root #mv /mnt/visionfive2/boot /mnt/visionfive2/mnt/sdcard/
root #ln -s /mnt/sdcard/boot /mnt/visionfive2/boot

Edit fstab to cater for any file systems that need to be mounted on the device, then unmount the image:

root #umount -R /mnt/visionfive2/mnt/sdcard
root #sync
root #losetup -d /dev/loop0

Write the image to a TF card

There are many methods available to do this. sys-boot/etcher-bin is a common method for those that want an easy-to-use GUI, however this example will use dd:

root #dd if=visionfive2.img | pv | dd of=/dev/mmcblk0 bs=4M

Move the secondary GPT to the actual end of the disk, delete the last partition then recreate it, and finally resize the file system:

root #sgdisk -e /dev/mmcblk0
root #sgdisk -d 4 /dev/mmcblk0
root #sgdisk --new=4:614400:0 --change-name=4:"root" --typecode=4:0FC63DAF-8483-4772-8E79-3D69D8477DE4 /dev/mmcblk0
root #partprobe /dev/mmcblk0
root #resize2fs /dev/mmcblk0p4

Boot the device

Once the TF card has been inserted into the device and the RGPIO switch is set to TF, the device should boot from the TF card.

Useful notes

Some useful notes that may be of interest to the reader can be found below.

Musl

This example uses a glib libc. It is possible to use musl libc as the systems C library. The TL;DR is:

  • Use the tuple riscv64-unknown-linux-musl instead of riscv64-unknown-linux-gnu wherever crossdev is in use.
  • Obtain (or build) any lp64d musl stage3 tarball and use that.
  • Select an appropriate musl profile.

Faster installation

Anywhere that QEMU-user is invoked to build a cross-arch package, using Portage within a chroot may be replaced with an external installation utilizing crossdev to cross-compile binaries and portage to install them into the image as follows:

root #riscv64-unknown-linux-gnu-emerge --ask sys-kernel/dracut
root #cd rootfs
root #ROOT=$PWD/ riscv64-unknown-linux-gnu-emerge --ask --usepkgonly --oneshot sys-kernel/dracut

It will be faster to cross-compile packages and install them into the image than to use QEMU-user to build them within the chroot, though this is not the preferred approach.

make.conf

Some useful additions for cross-compiling packages and identifying breakage in failed package builds:

FILE /etc/portage/make.conf
# Colour in portage output, useful for debugging
# Needed for ninja (e.g. z3)
CLICOLOR_FORCE=1
# https://gitlab.kitware.com/cmake/cmake/-/merge_requests/6747
# https://github.com/ninja-build/ninja/issues/174
CMAKE_COMPILER_COLOR_DIAGNOSTICS=ON
CMAKE_COLOR_DIAGNOSTICS=ON

# Common flags for cross-compiling and colour; params pulled from -march=native
COMMON_FLAGS="-mabi=lp64d -march=rv64imafdc_zicsr_zba_zbb -mcpu=sifive-u74 -mtune=sifive-7-series -O2 -pipe -fdiagnostics-color=always -frecord-gcc-switches --param l1-cache-size=32 --param l2-cache-size=2048"

# Enable QA messages for from iwdevtools
PORTAGE_ELOG_CLASSES="${PORTAGE_ELOG_CLASSES} qa"


See also

External resources