Android/SharkBait/Starting Android in LXC

After successfully booting Gentoo on Nexus 6P in the previous article, we can move on to launching Android in LXC, which is the last mission in the first period of my GSoC 2018 project. This article documents the process to bring up Android successfully with most of its functions working; only a few things don't work by now, and they're recorded in the section at the end of this article. Note that while this article strives to provide easy-to-follow instructions, it may not be suitable as a step-by-step tutorial. That is to say, you may encounter problems due to mistakes or things that the author didn't notice. The author appreciates your understanding when such things happen; meanwhile, feel free to contact the author by the means listed in the contact page.

Legends
Command prompts in this article starts with the hostname of the machine, followed by a  or , denoting if the user executing the command need to be root or not. The following hostnames have special meanings:


 * : the Linux workstation
 * : Gentoo Linux on Nexus 6P
 * : Android on Nexus 6P

Set Up Filesystem for Android
Android directly uses the initramfs as its rootfs and mounts other partitions, namely, , , , and , at mountpoints in. What we need to do here is to copy Android's rootfs to LXC's rootfs location so that we can launch it later on. Grab a copy of the (not the trimmed ), unpack it with the Makefile rules in  repository, tar it up, and transfer the tarball to the phone via adb. Remember to enable root access for ADB, which can be found in Developer Options in Android.

Copy the tarball into the Gentoo chroot and enter the chroot. Create the necessary mountpoints and unpack the root tarball into LXC rootfs.

We need  in Android rootfs to talk to the host for the  hack later on.

Write the following entries to   in Gentoo, so that OpenRC will take care of mounts on Gentoo startup.

Boot into Gentoo by temporarily booting the to check if the partitions are mounted correctly. Double-check that the device nodes are correct if things go wrong. Boot back into Android by rebooting.

Install LXC in Gentoo Linux
LXC is the core suite to boot Android containerized here. The installation process is in two parts: installing the userspace utilities, and enabling necessary options in the kernel.

Userspace Utilities
This is not different from installing normal Gentoo packages. Enter the chroot and install app-emulation/lxc:

The process can take a little bit of time to run. If it's too slow in your case, consider setting up a cross build environment per instructions on Gentoo Wiki if you have a Gentoo workstation. That's out of the scope of this article.

Kernel
According to the LXC article, we need quite many options for LXC's functioning. I've put how to compile a custom kernel in a separate article: Building a LXC-ready Android kernel with Gentoo toolchain. Follow that article to get a file, which is the kernel image that satisfies our needs, and proceed with the following instructions to repack the --we no longer need the  to be  from this point on.

The final should be located at. Boot the phone into, then flash the new image.

Boot into Gentoo by choosing &quot;Start&quot; in. Run to see if our kernel is suitable. Not everything's required; as long as you don't see red &quot;missing&quot; on items other than systemd, you're good to go.

Bring Android up in LXC container
Now that we have LXC correctly configured, we can set up the Android container. Some modifications to Andoid's and  are needed for a successful boot. Some hacks are required to fix some functions, of which we'll discuss in the next section. Commands in this section are expected to be executed with Gentoo booted up with, not in the chroot from Android.

The modified files' patches as well as helper scripts are put together in a repository on GitHub. Make sure that you apply them before continuing.

Place, , and in. Make sure that the executable bit is there for the scripts. Issue the following to see if LXC has recognized the container:

If everything's going as expected, start the container, and we'll see Android booting up on the device screen.

If the container failed to start, issue the following and then examine  to see what's wrong:

Verify that things are working, especially the hardware. In case something's crashing, or the Android framework failed to start at all, connect the phone to the computer via USB and check out via ADB (it starts pretty early). Tombstones at  in Android root are usually helpful; consult this article for how to read them.

Only some small bits don't work by now, and we'll cover them in the following section.

Container Auto-start
It's tiresome to attach a serial cable and issue to boot Android every time. Create service and add it to the  runlevel, as Android takes care of most of the device-specific work, namely networking:

Dial Home from Android world
Having to attach serial cable to access Gentoo is just too tiresome. Configure so that the Android guest can.

Enable in Gentoo, configure  for root, and put a little script  (available in in the  repository) for quick access:

We can now issue  to log into Gentoo. No more serial cable needed unless something goes wrong. We can safely disable ADB root access from Android Developer Settings by now.

Note that you can from Gentoo world! This will launch, which is the busybox sh. You may need to set to include   to get Android's utilities.

Fix Charger Mode
Android's init knows whether it's plugged into the charger while powered off. If that's the case, it launches, which draws the charging animation on the screen. Long-pressing the power button causes a restart, and the system will boot into normal Android. Unfortunately, our LXC stop hooks can't tell if the container ended with a poweroff or a restart, thus we won't be able to exit charger mode unless we unplug the power source.

The solution is quite simple: replace  with a simple script that places a marker file in   (which is also GNU/Linux's  ), and then call the real charger, which is now. The hook can then reboot if it sees that marker file. The script is available here.

How It Worked
If you take a look at the config for LXC, the start-stop hooks, and the patch, you'll discover that we needed to correct paths for cgroup pseudo-filesystems and set  RT schedule parameters. Android uses cgroups heavily as well, and we need to get it correct so that Android doesn't end up writing to the host's cgroup space. We need to set to allocate realtime bandwidth to the cgroup, otherwise realtime schedulers will fail. The following should be helpful to grab a general understanding of group scheduling:


 * Real-Time group scheduling
 * How do I configure LXC to allow the use of SCHED_RR in a container?

The rest of things (filesystem mounting, etc.) are pretty straightforward. Note that we need to tell Android ( to be specific) not to touch mounting, by commenting out > entries for the relevant partitions. Due to poorly-written code without  checks, we need an empty entry for   in Android, otherwise  will crash.

We bind  from GNU/Linux Android so that the container can talk to the host, informing the host about its states (such as &quot;We're in charger mode: reboot when I finish instead of power off&quot;). was chosen because it's the place to hold runtime files by convention on a GNU/Linux system. Read this article for more information.

Known Problems
The (non-exhaustive) list of problems as of 2018-05-26:


 * Reboot function in Android system (along with Advanced Reboot) doesn't work. This is because Android system doesn't actually poweroff the system or reboot it when inside a container: it simply performs, making it impossible for the GNU/Linux world (at least for now) to tell how the container stopped. The current behavior is that the system will power down regardless of whether &quot;Poweroff&quot; or &quot;Reboot&quot; (or other reboot options, e.g. Reboot to bootloader) has been selected in Android.
 * suffers from similar problems, though a workaround works pretty well; see The Charger section above.
 * This may be fixed by patching Android framework so that it signals the host about how it ended, by placing a file in the  filesystem bind-mounted from the host. The host then decides whether to reboot or to poweroff according to the signal file. This is exactly how 's problem gets fixed.
 * Boot is unbearably slow. Charging while powered off takes a long time to show the charging animation as well.
 * This is because the Linux kernel's initialization time is long. It takes over 10 seconds for the kernel to launch the first userspace program--.
 * For better experience, the kernel should have most of its functions as kernel modules, so that we can display a boot animation (or a static picture other than the Google logo from bootloader) as early as possible. We may draw an animation that's unrelated to in Android though.
 * Full-disk encryption / file-based encryption doesn't work.
 * This was done deliberately, as Android's implementation is a little bit too complicated to implement (see my previous analysis on Android).
 * Standard encryption means should get implemented at some point (e.g. LUKS).