Banana Pi the Gentoo Way

Preamble: This guide is going to provide details on how to install Gentoo with a user compiled kernel and Gentoo Stage 3. I have been using a number of Banana Pi's in a clustered distcc cross compiling environment for nearly a year.There has been a lot of controversy about the Banana Pi and it developers. The hardware is a nice board, but some of the practices concerning the methods undertook by the developers concerning business practices and the software side is certainly controversial. Best summed up on the Linux-Sunxi.org/Banana_Pi page.

To provide some clarification on documentation from different sources. Nearly everything provided on the Lemaker websites, and other Banana Pi websites has been copied from the Linux-Sunxi.org pages. While all the direct copies of the Linux-Sunxi.org website have been taken down. I am assuming some copyright infringement issues surfaced. For some reason, Lemaker prefers to maintain their own full kernel git repositories rather than simply having a simple patch repository and providing patches to upstream Linux-Sunxi.org and U-Boot to be included in their respective git repositories. This practice makes it more difficult for LeMaker patches to be applied to other kernel sources by the general user base. Some early patches were provided and are included in the Linux-Sunxi.org git repositories, but latter refinements have not been submitted for inclusion.

I have to provide some cautionary advice concerning some of the ready to go images for the Banana Pi, along with some of the git repositories. The LeMaker Gentoo release is not put together very well, specifically when it comes to CFLAGS. It is obvious that they are inexperienced and are not aware of the impact of having incorrect or missing CFLAGS. I have seen countless posts of people spending countless hours attempting to get hardware acceleration working, while the CFLAGS supplied with the LeMaker Gentoo release, will not allow hardware acceleration to its fullest. At least it did not earlier this year 2015. Here is a post of mine concerning CFLAGS for the Banana Pi.

I have found numerous posts on a number of forums, as well as the Lemaker forums themselves, with claims of malicious content. I my self have not bothered with disseminating any of the so-called official releases, and cannot confirm any of the claims. Considering the clashes and legal disputes over trademarks and domain ownership, along with the claim of open source, when the Banana Pi is certainly not, visit the Linux-Sunxi.org website for more details on that. It may certainly be true that some parties may have injected malicious content in an attempt to harm other participating parties.

The best way to avoid any possible security risks, is to use software that has been reviewed and tested by a reputable process and group. Therefore this guide will provide details on how to compile a kernel from the Linux-Sunxi.org git repository, who is an open source software community dedicated to providing open source operating system support for Allwinner SoC based devices. Instructions for using the Banana Pi (Lemaker) developers git repository are also included. The Operating System Source files will be a standard Gentoo Arm 7 Stage 3 pulled from an Approved Gentoo Mirror.

To speed up compilation, it is suggested that you set up a distcc cross compiling environment on an Amd64 or other faster architecture. Distcc and Distcc/Cross-Compiling. Not all packages will compile on other machines in the cluster, but many will. If you have a number of Banana Pi devices, use the buildpkg feature so that binary packages are built that can then be quickly installed on the other Banana Pi's or distributed to devices in other locations. An NFS mount can be used to have the packages directory mounted on all the Banana Pi devices on your lan, so that binary packages are available to all devices without having to manually distribute the binary packages.

Intro to the Banana Pi
The Banana Pi has a Cortex-A7 Arm processor Model A20 manufactured by Allwinner Technology. Which is a different architecture than what most common place computers have. Most desktops and laptops are either the newer amd64 or the older x86. It is not possible to compile code with one gcc architecture for another. Therefore to compile a kernel for the Banana Pi, if you do not already have an Arm 7 device running, you must setup a Cross Compiling environment on an amd64 or x86 computer. Fortunately it is a relatively simple thing to do. I highly suggest everyone at least once, go through the procedure as it is not all that difficult, what you will learn will provide for a great building block for the future.

If you just want to get up and running with out all the compiling. I am providing the 4 necessary files already compiled and tested out. These files including the kernel have been configured and compiled from the Linux-Sunx.org git repository

If you download the prepared files you can skip much of the instruction and only need to do the below sections.

Preparing The SDCard
Insert your sdcard and run lsblk.

Look for your sdcard, you should see a list of all your drives and mounted partitions with their attachment points starting with sda. I will be using /dev/sdX with X being used as the letter for for your sdcard.

This will clear all the data in the first part of the sdcard to ensure no old data causes any problems.

You need a minimum of two partitions. 1. Boot - 75Mb - 150Mb will do, perhaps larger if you intend on storing a lot of different kernels or other files. 2. Root - Can be the rest of the disk, or you may make a home partition, var partition or can even do a swap partition. Any partition configuation you want.

At the fdisk prompt, delete old partitions and create a new one: a.Type o. This will clear out any partitions on the drive. b.Type p press enter to list partitions. There should be no partitions left. c.Type n press enter, then p press enter for primary, 1 for the first partition on the sdcard/drive press enter, 2048 for the first sector press enter. d.Type +150M and press enter. e.Type p and you will see the first partition listed. f.Type n press enter, then p press enter for primary, 2 for the second partition on the sdcard/drive press enter, press enter to select the default, should be 309248 if you did a 150MB first partion, if you press enter the default will be for the rest of the space left on    the disk. If you plan on additional partitions select the size you want for your second partition. g.Type p press enter and you should see two partions, first one 150M the second one the rest of your sdcard/disk. h.Type t press enter, type 1 for the first partition press enter, type l press enter, you will see a list, press b and press enter to    select W95 FAT32. i.Type p press enter, you should now see two partitions with the first as W95 FAT32 and the second as Linux. Linux does not need a boot partition toggled as bootable, you may do so if you wish, as it will not hurt anything, but it is absolutely unnecessary. j. Type w press enter to write the partition data to the sdcard/disk.

Now we are going to format the partitions with a file system.

Now mount the partitions so the boot files and system files can be loaded after we complete preparing and building them.

If you downloaded the pre-compiled files place them in /mnt/bananapi/boot

Install Root File System - Arm 7 Stage 3
Go to the mirrors page and download a stage3-armv7a_hardfp-latest.tar.bz2 or with most recent date stage3-armv7a_hardfp-20150721.tar.bz2 Gentoo Mirrors Get what ever the latest build is, it may be different than shown below:

If using the pre-compiled kernel skip to: Final Boot Files

Cross Compiling
If you do not have a running arm device/computer to build from, and are running an Amd64 or x86 Desptop/Laptop. You will need to setup a cross compiling environment. Follow the Distcc/Cross-Compiling guide. You may also want to review the Raspberry Pi Cross building & BealgeBone Black. Since compiling on one Banana Pi is not all that speedy you will want to take advantage of faster computers on you network. Distcc

When you install your arm crosdev it is important that you use the cflags that you will be using on the Banana Pi. If neon-vfpv4 is not used, you will be losing some graphics performance capability. While you are compiling on your Cross-Compiler, you can watch the console feed for warnings and errors, if EXTRA_CONF settings are incorrect or missing, you will see warnings, though the compilation in most cases will continue.

Here is a link to some information: Arm Cortex A Processors & GCC Commands.

You will need to emerge and install crossdev and will need git for downloading git repositories.

You will need to run this command on every machine that is not an Arm device that will be used for compiling and or distcc.

DistCC/Cross-Compiling
Distcc will speed up compiliation of packages by distributing the compiling to one or more computers in your lan. Not all programs compiled on the Bananan Pi will finish compiling with distcc enabled. With a few programs it will be necessary to set your set your MAKEOPTS to MAKEOPTS="-j1 -l2", which will keep all compilation on the single device. If a package fails when compiling, the first thing to try is change your MAKEOPTS.

Every computer that you want to use will have to have Distcc installed as well as Crossdev if is is not an Arm 7 device.

Install Distcc:

You need to add the features setting to the make.conf file on all computers that are going to be used for distcc:

If you want to use distcc-pump do the following. Reference the distcc documentation for details on distcc-pump:

You will now have to set the distcc hosts in the /etc/distcc/hosts file:

You have to configure the /etc/conf.d/distcc file on all your computers that you want listening to receive distcc orders from the Banana Pi as well as on the Banana Pi:

Start Distcc and add to rc-update to be started on boot on all machines you want distcc to be running on:

If you have Iptables running you will need to open the distcc ports on all machines:

You will need to make some changes on the Banana Pi so the computers that are recieving distcc orders from the Banana Pi use the correct gcc version: Referance the Distcc/Cross-Compiling documentation:

You will need to replace the links with a link to a wrapper that will tell the build machines which version of gcc to use:

Double check that you did things right:

You can monitor distcc a number of ways:

If you have a window manager or desktop running you can use the gui

To monitor Portage's distcc usage:

If you use Conky you can see distcc being used in inbound and outbound connections. Conky can be configured to display distcc details via the distccmon monitor or display activity by portage and or activity in the /var/tmp/portage/.distcc folder.

U-Boot
You will need to build a bootloader specific for the Banana Pi. You have a choice of using either the Linux-Sunxi git repositories or the Lemaker git repository. If you prefer to us the Linux-Sunxi repository you can simply download the boards.cfg from LeMaker to see if it has been updated and not provided to upstream Linux-Sunxi. You can simply edit the changes or copy over the LeMaker boards.cfg. You can visit the git repository to check without having to clone both repositories. With LeMaker being in such a disarray, there is no telling who has access to their repositories, therefore I stay away from them. I have included the links to the Lemaker repository for those that still want to use them.

If you do not have u-boot-tools installed install it:

Make a build folder to work in. It can be called anything and can be located anywhere you want. Most things can be configured as a regular user, but then have to be compiled as root.

Since both repositories are named the same, you will need to keep them separated:

Clone the Sunxi git repository:

Change to the LeMaker folder:

Clone the LeMaker

Check for differance:

If they are different you can either append the sunxi boards.cfg or make a copy and over-write it.

To use Sunxi:

To use LeMaker:

Then run the following commands in either repository - First you need to configure for the Banana Pi:

If you have the Banana Pi Pro

Now you need to compile the bootloader:

Lets put the booatloader in the boot partition. We do not have to, you can keep it anywhere, but it is nice to have it there, as it will always be with the device if it ever needs to be reloaded.

Burn the bootloader into the SD card:

Script.bin
You will now need to make the script.bin file. Once again you can use either the Linux-Sunxi or the LeMaker git repository. To make sure you are using the latest updated .fex file if using Linux-Sunxi get BananPi.fex or BananaPro.fex from the LeMaker repository.

For Sunxi:

For LeMaker:

To check for differences Banana Pi:

To check for differences Banana Pro

You can either append or copy over the Lemaker fex file to the Sunxi folder:

To build with Sunxi:

To build with LeMaker:

You will now need to make a change to the BananaPi.fex and BananaPro.fex files. No matter which repository you decide to use. Open the file and search for "ctp_det_used":

For the Banana Pi

For the BananaPro

Compile the Kernel
To get started with a Sunxi kernel configuration use the sun7i_defconfig. Download the sun7i_lemaker_defconfig which has more things selected, both are a good starting points depending on what all you want configured in the kernel.

To use the sun71_lemaker_defconfig

Here is my custom config, some unnecessary things are turned off, while iptables and NFS are on as well as a few other common things were compiled into the kernel. Everything I needed is compiled into the kernel. There are a lot of modules selected, did not spend the time to go through and de-select every unneeded module. Simply did not compile the modules as they are not needed for my needs. custom_bananapi_defconfig

There are a lot of things with the LeMaker config selected that are only needed if you intend on attaching various devices to the board. For a smaller kernel go ahead de-select the devices you will not be needing, and or add devices you will be using or other features/modules you want. You only need to modify one, or use one as is.

If you prefer to use a GUI for configuring the kernel use one of the below:

First load the bananapi kernel configuration choose only one!

Tweak the kernel configuration: Change nconfig to which ever is your favorite gui in the list above:

Compile the Kernel:

Create Modules: Creating modules is not necessary if necessary features are compiled directly into the kernel, as with my custom config.

After the Kernel is finished compiling:

Copy the modules over to the root file system: Replace XX with what ever the kernel number is: The modules will be placed in output/lib/moduels/3.4.XX the xx being what ever version the kernel has been bumped to, when the kernel and modules complete the version will be displayed:

Final Boot File
You will need to create a uEnv.txt file that will contain specific directions for the boot process:

These settings have NO Memory Reserve. Which works fine for a headless setup. If you are going to be running a window manager or desktop, you will want to reserve some memory. Here is the link to the Linux-Sunxi Kernel Arguments Page for some additional settings. What ever amount of memory is reserved, will not be available for general system use.

Booting to an SATA SSD Disk
Here are some hdparm results to show you why you want to use an ssd.

sdcard: root@bananapi ~ # hdparm -Tt /dev/mmcblk0

/dev/mmcblk0: Timing cached reads:  726 MB in  2.00 seconds = 362.66 MB/sec Timing buffered disk reads: 48 MB in  3.02 seconds =  15.91 MB/sec

root@bananapi ~ # hdparm -Tt /dev/sda

/dev/sda: Timing cached reads:  784 MB in  2.00 seconds = 391.21 MB/sec Timing buffered disk reads: 374 MB in 3.00 seconds = 124.60 MB/sec

It is not only the overall MB/sec but even more importantly the small file rate. Bonnie++, iozone, piozone tests have consistently shown on numerous arm devices that 4k,8k,16k read writes are much slower with an sdcard. A $100 150MB/sec read capable sdcard will be NO faster. Next in speed is usb disk and internal emmc for those boards that have them. With SATA SSD far outstripping anything else.

Finalizing in preparation for booting
You will need to edit the make.conf and specifically set the correct CFLAGS & GCC-Optimization, or certain functionality will not be possible. Use Flags This is my make.conf, take note of the MAKEOPTS, and other settings specific to Distcc, you need to set some depending on whether you are using Distcc and how many total cpu's are avaialable, reference the Distcc & Distcc/Cross-Compiling documentation for details. Unless you intend upon using the Overlay & Layman packages, you should leave those commented out. Gentoo is very flexible, spend some time learning some of the things that can be done with it, you will eventually figure out what you need and do not need in the make.conf. These make.conf settings have been very stable and efficient. Two Banana Pi devices withstood six months of pressure testing with various server configurations.

Here are some links to some details that might help you decide what the best CFLAGS may be. Arm Cortex Processors & GCC Neon DSP & SIMD Neon Development Article

Edit /mnt/bananapi/etc/shadow so root can login:

Get a hash output for the root password:

Optional
If you are not going to have keyboard access and need access via SSH. dhcpcd is not installed so a static address will have to be set: edit

The bare minimum will work for just getting in via ssh: edit

Edit /mnt/bananapi/root/etc/conf.d/hostname:

Add net.eth0 to startup:

Add sshd to startup:

Replace hwclock with swclock. So the OS gets a time on boot as it has no hardware clock.

Set timezone - This will have to be set after your first boot.

Plug in your sdcard and boot
If all went well you should get a boot prompt.

Tips & Tricks & Uses
NFS can be used to distribute binary packages to all Banana Pi devices on the network, or other Arm 7 devices that use the same CFLAGS. The NFS server can be a Banana Pi or any other computer in your lan. Preferably another computer as the 1Gb nic is really only capable of 500-600Mb/sec. Bonding can be used on your NFS server to help with good throughput. A one time setup of an NFS share to mount as your package directory will allow all devices to pull the binary packages without any manual distribution or intervention.

The Banana Pi works great for running UniFi Wi-Fi Device Management Software for Ubiquiti Wireless Products. The Banana Pi is powerful enough to handle multiple AP'S. Perfect for homes or small business's that want an inexpensive solution to a self contained Enterprise level wireless network. Access is via browser from any computer on the lan or wan.