Android/SharkBait/Starting Android in LXC

Preface
After successfully booting Gentoo on Nexus 6P in the previous article, we can move on to launching Android in LXC, which is the last mission in the first period of my GSoC 2018 project. This article documents the process to bring up Android successfully with most of its functions working; only a few things don't work by now, and they're recorded in the Known Problems section at the end of this article. Note that while this article strives to provide easy-to-follow instructions, it may not be suitable as a step-by-step tutorial. That is to say, you may encounter problems due to mistakes or things that the author didn't notice. The author appreciates your understanding when such things happen; meanwhile, feel free to contact the author by the means listed in the page linked at the bottom of this page.

Legends
Command prompts in this article starts with the hostname of the machine, followed by a  or , denoting if the user executing the command need to be root or not. The following hostnames have special meanings:


 * : the Linux workstation
 * : Gentoo Linux on Nexus 6P
 * : Android on Nexus 6P

Set Up Filesystem for Android
Android directly uses the initramfs as its rootfs and mounts other partitions, namely,  ,  ,  , and  , at mountpoints in. What we need to do here is to copy Android's rootfs to LXC's rootfs location so that we can launch it later on. Grab a copy of the Android  (not the trimmed    ), unpack it with the Makefile rules in   repository, tar it up, and transfer the tarball to the phone via adb. Remember to enable root access for ADB, which can be found in Developer Options in Android.

yuki $ adb devices List of devices attached 84xxxxxxxxxxxxxx       device

yuki $ adb root yuki $ adb shell dd if=/dev/block/bootdevice/by-name/boot | dd of=boot-stock.img ( ... output elided ... ) yuki $ mv boot-stock.img ~/preinit_angler/initramfs/boot.img &amp;&amp; cd ~/preinit_angler yuki $ make unpack ( ... output elided ... ) yuki $ cd initramfs/root &amp;&amp; tar czvf -. | adb shell &quot;cat &gt;/data/android-root.tar.gz&quot; yuki $ Copy the tarball into the Gentoo chroot and enter the chroot. Create the necessary mountpoints and unpack the root tarball into LXC rootfs.

yuki $ adb shell angler # /data/gnu/start_chroot angler # mv /data/android-root.tar.gz /data/gnu/root/ angler # chroot /data/gnu /bin/su umi # mkdir -p /var/lib/android/{system,vendor,data,system,cache} umi # mkdir -p /var/lib/lxc/android/rootfs umi # tar xvf ~/android-root.tar.gz -C /var/lib/lxc/android/rootfs ( ... output elided ... ) umi # mkdir -p /var/lib/lxc/android/run umi # We need  in Android rootfs to talk to the host for the   hack later on.

Write the following  entries to   in Gentoo, so that OpenRC will take care of mounts on Gentoo startup.

/dev/mmcblk0p43 /var/lib/android/system ext4 ro,barrier=1,inode_readahead_blks=8 0 0 /dev/mmcblk0p32 /var/lib/android/persist ext4 noatime,nosuid,nodev,barrier=1,data=ordered,nomblk_io_submit 0 0 /dev/mmcblk0p37 /var/lib/android/vendor ext4 ro,barrier=1,inode_readahead_blks=8 0 0 /dev/mmcblk0p38 /var/lib/android/cache ext4 noatime,nosuid,nodev,barrier=1,data=ordered,nomblk_io_submit,noauto_da_alloc 0 0 /dev/mmcblk0p44 /var/lib/android/data ext4 noatime,nosuid,nodev,barrier=1,data=ordered,nomblk_io_submit,noauto_da_alloc,inode_readahead_blks=8 0 0
 * 1) Android mounts

/var/lib/android/system /var/lib/lxc/android/rootfs/system none bind 0 0 /var/lib/android/persist /var/lib/lxc/android/rootfs/persist none bind 0 0 /var/lib/android/vendor /var/lib/lxc/android/rootfs/vendor none bind 0 0 /var/lib/android/cache /var/lib/lxc/android/rootfs/cache none bind 0 0 /var/lib/android/data /var/lib/lxc/android/rootfs/data none bind 0 0 /run /var/lib/lxc/android/rootfs/run none bind 0 0 Boot into Gentoo by temporarily booting the    to check if the partitions are mounted correctly. Double-check that the device nodes are correct if things go wrong. Boot back into Android by rebooting.
 * 1) Bind into container

Install LXC in Gentoo Linux
LXC is the core suite to boot Android containerized here. The installation process is in two parts: installing the userspace utilities, and enabling necessary options in the kernel.

Userspace Utilities
This is not different from installing normal Gentoo packages. Enter the chroot and install :

yuki $ adb root yuki $ adb shell /data/gnu/start_chroot yuki $ adb shell chroot /data/gnu /bin/su umi # emerge --verbose --autounmask-write=y app-emulation/lxc ( ... output elided ... ) umi # dispatch-conf ( ... output elided ... ) umi # emerge --verbose app-emulation/lxc ( ... output elided ... ) umi # The process can take a little bit of time to run. If it's too slow in your case, consider setting up a cross build environment per instructions on Gentoo Wiki if you have a Gentoo workstation. That's out of the scope of this article.

Kernel
According to LXC on Gentoo Wiki, we need quite many options for LXC's functioning. I've put how to compile a custom kernel in a separate article: Building a LXC-ready Android kernel with Gentoo toolchain. Follow that article to get a  file, which is the kernel image that satisfies our needs, and proceed with the following instructions to repack the    --we no longer need the Android   to be   from this point on.

yuki $ cp Image.gz-dtb ~/preinit_angler/out/zImage &amp;&amp; cd ~/preinit_angler yuki $ make repack ( ... output elided ... ) yuki $ The final    should be located at. Boot the phone into, then flash the new image.

yuki $ fastboot devices 84xxxxxxxxxxxxxx       fastboot yuki $ fastboot flash boot out/boot-mod.img ( ... output elided ... ) yuki $ Boot into Gentoo by choosing &quot;Start&quot; in. Run  to see if our kernel is suitable. Not everything's required; as long as you don't see red &quot;missing&quot; on items other than systemd, you're good to go.

umi # lxc-checkconfig --- Namespaces --- Namespaces: enabled Utsname namespace: enabled Ipc namespace: enabled Pid namespace: enabled User namespace: enabled Network namespace: enabled Multiple /dev/pts instances: enabled

--- Control groups --- Cgroups: enabled

Cgroup v1 mount points: /sys/fs/cgroup/openrc /sys/fs/cgroup/cpuset /sys/fs/cgroup/debug /sys/fs/cgroup/cpu /sys/fs/cgroup/cpuacct /sys/fs/cgroup/devices /sys/fs/cgroup/freezer

Cgroup v2 mount points:

Cgroup v1 systemd controller: missing Cgroup v1 clone_children flag: enabled Cgroup device: enabled Cgroup sched: enabled Cgroup cpu account: enabled Cgroup memory controller: missing Cgroup cpuset: enabled

--- Misc --- Veth pair device: enabled, not loaded Macvlan: enabled, not loaded Vlan: missing Bridges: enabled, not loaded Advanced netfilter: enabled, not loaded CONFIG_NF_NAT_IPV4: enabled, not loaded CONFIG_NF_NAT_IPV6: missing CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded CONFIG_IP6_NF_TARGET_MASQUERADE: missing CONFIG_NETFILTER_XT_TARGET_CHECKSUM: missing CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore --- checkpoint restore: missing CONFIG_FHANDLE: enabled CONFIG_EVENTFD: enabled CONFIG_EPOLL: enabled CONFIG_UNIX_DIAG: enabled CONFIG_INET_DIAG: enabled CONFIG_PACKET_DIAG: enabled CONFIG_NETLINK_DIAG: enabled File capabilities:

Note : Before booting a new kernel, you can check its configuration usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

umi #

Bring Android up in LXC container
Now that we have LXC correctly configured, we can set up the Android container. Some modifications to Andoid's  and   are needed for a successful boot. Some hacks are required to fix some functions, of which we'll discuss in the next section. Commands in this section are expected to be executed with Gentoo booted up with, not in the chroot from Android.

The modified files' patches as well as helper scripts are put together in a repository on GitHub. Make sure that you apply them before continuing.

Place,  , and   in. Make sure that the executable bit is there for the scripts. Issue the following to see if LXC has recognized the container:

umi # lxc-info android Name:          android State:         STOPPED If everything's going as expected, start the container, and we'll see Android booting up on the device screen.

umi # lxc-start android umi # If the container failed to start, issue the following and then examine  to see what's wrong:

umi # lxc-start -l debug -o start.log -n android ( ... error output elided ... ) umi # Verify that things are working, especially the hardware. In case something's crashing, or the Android framework failed to start at all, connect the phone to the computer via USB and check out  via ADB (it starts pretty early). Tombstones at  in Android root are usually helpful; consult this article for how to read them.

yuki $ adb logcat | less yuki $ adb shell angler # cd /data/tombstones Only some small bits don't work by now, and we'll cover them in the following section.

Container Auto-start
It's tiresome to attach a serial cable and issue  to boot Android every time. Create  service and add it to the   runlevel, as Android takes care of most of the device-specific work, namely networking:

umi # cd /etc/init.d umi # ln -s lxc lxc.android umi # rc-update add lxc.android boot

Dial Home from Android world
Having to attach serial cable to access Gentoo is just too tiresome. Configure  so that the Android guest can.

angler # ssh-keygen -t ed25519 -C android -f /data/ssh/id_ed25519 Enable  in Gentoo, configure   for root, and put a little script   (available in in the   repository) for quick access:

umi # umask 077 umi # mkdir -p ~/.ssh umi # cat /var/lib/android/data/ssh/id_ed25519.pub &gt;&gt; ~/.ssh/authorized_keys umi # cp dialhome /var/lib/android/data/ssh/ umi # chmod u+x /var/lib/android/data/ssh/dialhome umi # chown 2000:2000 /var/lib/android/data/ssh/dialhome umi # rc-update add sshd default We can now issue  to log into Gentoo. No more serial cable needed unless something goes wrong. We can safely disable ADB root access from Android Developer Settings by now.

Note that you can  from Gentoo world! This will launch, which is the busybox sh. You may need to set  to include   to get Android's utilities.

Fix Charger Mode
Android's init knows whether it's plugged into the charger while powered off. If that's the case, it launches, which draws the charging animation on the screen. Long-pressing the power button causes a restart, and the system will boot into normal Android. Unfortunately, our LXC stop hooks can't tell if the container ended with a poweroff or a restart, thus we won't be able to exit charger mode unless we unplug the power source.

The solution is quite simple: replace  with a simple script that places a marker file in   (which is also GNU/Linux's  ), and then call the real charger, which is now. The  hook can then reboot if it sees that marker file. The script is available here.

How It Worked
If you take a look at the config for LXC, the start-stop hooks, and the  patch, you'll discover that we needed to correct paths for cgroup pseudo-filesystems and set   RT schedule parameters. Android uses cgroups heavily as well, and we need to get it correct so that Android doesn't end up writing to the host's cgroup space. We need to set  to allocate realtime bandwidth to the cgroup, otherwise realtime schedulers will fail. The following should be helpful to grab a general understanding of group scheduling:


 * Real-Time group scheduling
 * How do I configure LXC to allow the use of SCHED_RR in a container?

The rest of things (filesystem mounting, etc.) are pretty straightforward. Note that we need to tell Android ( to be specific) not to touch mounting, by commenting out   entries for the relevant partitions. Due to poorly-written  code without   checks, we need an empty entry for   in Android , otherwise   will crash.

We bind  from GNU/Linux Android so that the container can talk to the host, informing the host about its states (such as &quot;We're in charger mode: reboot when I finish instead of power off&quot;). was chosen because it's the place to hold runtime files by convention on a GNU/Linux system. Read this article for more information.

Known Problems
The (non-exhaustive) list of problems as of 2018-05-26:


 * Reboot function in Android system (along with Advanced Reboot) doesn't work. This is because Android system doesn't actually poweroff the system or reboot it when inside a container: it simply performs, making it impossible for the GNU/Linux world (at least for now) to tell how the container stopped. The current behavior is that the system will power down regardless of whether &quot;Poweroff&quot; or &quot;Reboot&quot; (or other reboot options, e.g. Reboot to bootloader) has been selected in Android.
 * suffers from similar problems, though a workaround works pretty well; see The Charger section above.
 * This may be fixed by patching Android framework so that it signals the host about how it ended, by placing a file in the  filesystem bind-mounted from the host. The host then decides whether to reboot or to poweroff according to the signal file. This is exactly how  's problem gets fixed.
 * Boot is unbearably slow. Charging while powered off takes a long time to show the charging animation as well.
 * This is because the Linux kernel's initialization time is long. It takes over 10 seconds for the kernel to launch the first userspace program--.
 * For better experience, the kernel should have most of its functions as kernel modules, so that we can display a boot animation (or a static picture other than the Google logo from bootloader) as early as possible. We may draw an animation that's unrelated to  in Android though.
 * Full-disk encryption / file-based encryption doesn't work.
 * This was done deliberately, as Android's  implementation is a little bit too complicated to implement (see my previous analysis on Android cryptfs).
 * Standard encryption means should get implemented at some point (e.g. LUKS).
 * Choosing to encrypt device in System Settings in Android can cause unexpected behaviors. YOU HAVE BEEN WARNED.