Hardened/PaX Quickstart

Understanding and working with PaX in Hardened Gentoo.

What is Hardened Gentoo?
The Hardened Gentoo project focuses on adding features to a Gentoo system that help resist security compromises. The approach is not always systematic, and many features that enhances security are added. Some of these compliment one another (e.g. PIE from toolchain hardening and ASLR from PaX kernel hardening), and some are mutually exclusive (e.g. SELinux and Grsecurity's RSBAC kernel hardening). This document focuses on  which adds security enhancement to that area between both kernel and user land.

What is PaX?
PaX is a patch to the Linux kernel that provides hardening in three ways.

1. The first protection provided by PaX is a judicious enforcement of non-executable memory. This prevents a common form of attack where executable code is inserted into the address space of a process by an attacker and then triggered, thus hijacking the process and possibly escalating privileges. The typical vector for the insertion is via user provided data that finds its way into executable memory. By ensuring that "data" only lives in memory which is non-executable, and that only "text" is found in memory which is executable, PaX pre-emptively protects against this class of attacks.

Try running the following with PaX's MPROTECT enforced and then not enforced to see this feature in action:

mmap-rwx.c: violate MPROTECT with RWX mmap

Similarly, try running the following with EMULTRAMP both enabled and disabled. This convoluted code forces gcc to set up a trampoline for the nested function f2. The trampoline is executable code which lives on the stack and could allow the user to inject their own code via the variable i.

trampoline.c: force gcc to generate a trampoline

2. The second is Address Space Layout Randomization (ASLR). This provides a randomization of the memory map of a process (as reported, for example, by pmap) and thus makes it harder for an attacker to find the exploitable code within that space. Each time a process is spawned from a particular ELF executable, its memory map is different. Thus exploitable code which may live at 0x00007fff5f281000 for one running instance of an executable may find itself at 0x00007f4246b5b000 for another. While the vanilla kernel does provide some ASLR, a PaX patched kernel increases that. Furthermore, when an application is built as a Position Independent Executable, even the base address is randomized. Try repeatedly running the following on both a vanilla and PaX kernel, with and without PIE:

aslr-test.c: randomize base addess for PIE

For more information on PIE, see our documentation on the Hardened/Toolchain.

3. Finally, the PaX patches provide some miscellaneous hardening: erasing the stack frame when returning form a system call, refusing to dereference user-land pointers in some contexts, detecting overflows of certain reference counters, correcting overflows of some integer counters, enforcing the size on copies between kernel and user land, and providing extra entropy.

More information on PaX can be found on its official homepage.

Understanding PaX
The first step in working with PaX is to configure and boot a PaX patched kernel. Depending on whether or not you've configured PaX for SOFTMODE or non-SOFTMODE, the kernel will automatically start enforcing memory restrictions and address space randomization on all running processes. Ideally you shouldn't have to do anything else; however, for problematic executables in non-SOFTMODE, a second step is required: you may have to relax certain PaX restrictions on a per ELF object basis. This is done by tweaking the PaX flags which are read by the kernel when the ELF is loaded into memory and execution begins. This second step is usually straight forward except when the ELF object that requires the relaxation is a library. In that case, the library's flags have to be back ported to the executable that links against it because when PaX enforces/relaxes its features, it does so on the basis of the executable's flags, not those of the libraries it links against. We will discuss both steps in detail below, but first we need an overview of PaX's features. We'll summarize these here, and for more details, we refer the reader to the official PaX documentation.

SOFTMODE: If this option is selected, PaX protection will not be enforced by default for those features which can be turned on or off at runtime, so this is the "permit by default" mode in contrast to the "forbid by default" which is obtained when this option is not selected. In SOFTMODE, the user must explicitly mark executables to enforce PaX protections, whereas in non-SOFTMODE, the user must explicitly mark to relax PaX protections.

Enforce non-executable pages:

Enhanced Address Space Layout Randomization (ASLR):

Miscellaneous Memory Protection:

As stated above, some PaX features can be enforced (in the case of SOFTMODE) or relaxed (in the case of non-SOFTMODE) on a "per ELF object" basis. These are PAGEEXEC, EMULTRAP, MPROTECT, RANDMMAP and SEGMEXEC and these are respectively controlled by the following flags: P, E, M, R, S and p, e, m, r, s. The upper case mean "enforce" in SOFTMODE and the lower case mean "relax" in non-SOFTMODE. A third possibility is that neither the enforce or relax flags are set, in which case the kernel simply uses the default for that particular protection. E.g. if neither P nor p is set on an ELF object, then the kernel will not enforce PAGEEXEC in SOFTMODE and will enforce it in non-SOFTMODE.

Currently the PaX patches support three ways of doing PaX markings: EI_PAX, PT_PAX and XATTR_PAX. EI_PAX places the PaX flags in bytes 14 and 15 of the e_ident field of an ELF objects's header. But this is broken in recent versions of glibc and is no longer supported. (See ) PT_PAX places the flags in a ELF objects's program header called PAX_FLAGS. This has the advantage the that flags are in the body of the object and will always be carried with it when copied; but, it has the disadvantage that the object must have th PAX_FLAGS program header to work. Most Linux distributions don't build their executables and libraries with this program header, and adding it is problematic: there may not be enough space in the ELF object to add it or converting a GNU_STACK program header, which is not used by the PaX kernel, might later cause problems under other kernels if the object is exported. In the worst case, changing the ELF binary will break self-checking executables which detect that they have been "tampered" with. (See ).

While PT_PAX is still supported, the preferred approach is to use XATTR_PAX which places the PaX flags in a file system's extended attributes. This has the advantage that it does not modify the ELF object in any way, but the disadvantage that the filesystems which house these objects, and the utilities used to copy, move and archive, them, must support xattrs. In the case of Gentoo and portage, this means that tmpfs must support the user.pax.* xattr namespace in which the PaX flags are placed, not just the security.* and trusted.* namespaces. (Note: user.pax.* is the subspace of user.* which will be used for PaX related xattr information. We do not enable the entire user.* namespace in tmpfs to reduce the risk of attack vectors via that path.)

One final caveat about the two supported methods of doing PaX markings: the PaX kernel allows you to enable both PT_PAX and XATTR_PAX, but if you do so, it will not impose the markings unless the same flags are found in both locations. For this reason, we do not recommend enabling both, even for migration which we describe below.

Building a PaX Kernel
The Hardened Gentoo team maintains and supports the  package, which we will cover here. Other in tree kernel sources may have PaX, and much of what we say may apply, but you will have to work through the differences yourself. The hardened-sources come with the Grsecurity patches (http://grsecurity.net/), which bundle the PaX patches. If you want only the PaX patches, these can be obtained in isolation from http://www.grsecurity.net/~paxguy1/. If you are interested in learning more about Grsecurity hardening in general, we cover that in our Grsecurity Quickstart.

Emerging a hardened-sources kernel places the kernel source tree at, but it does not configure it for you. It is now up to you to make sure PaX is configured so that it enforces or relaxes what you want. Below we will concentrate on recommended configurations, but feel free to deviate. The previous section should have given you some idea as to what each of the options provide so you can make intelligent choices. We recommend tarting off with as much hardening as possible and relaxing only when there is no other workaround.

As stated, the PaX patches are bundled with Grsecurity, so the PaX configuration options are found under that menu in. You also have the option of selecting one of Grsecurity's preconfigured profiles at. These will give you a meaningful starting point configuration for PaX.

If configuring for only PT_PAX, the following should be sufficient. Note that since we are trying to show the configuration options for both x86 and amd64, we've marked where the differences lie with a  in the left most column.

Please note that these are recommended settings and you can deviate from them, but be careful when doing so. On a play box, you can and should try new features to see what they might do or break. But on a production box, do not turn on or off some feature without knowing what you are doing. This is particularly true for EMUTRAMP (Emulate trampolines) which is turn on by default, and turning it off can break python and therfore your system! The reason is that python and libffi write code on the fly, either by RWX mmap-ings which PaX's MPROTECT will kill, or by falling back on trying to write little code snippets to files which other features in Grsecurity will kill, like Trused Path Execution. (See bug #484472.) In Gentoo, we've opted to patch python and libffi to use trampolines that can be safely handled by emulation in a PaX kernel. (See bug #329499.) We then PaX mark the python executable with 'E' to enable emulation of trampolines. Without EMUTRAMP on in the kernel, python will fail in trying to generate its RWX mmap-ings and die every time. (As a temporary workaround if you get into this situaiton, you can disable MPROTECT on python with `paxctl -m` and then restore the original markings once you have booted back into a PaX kernel with EMUTRAMP.)

Also, we should prefer XATTR_PAX flags over PT_PAX. In that case, all of the above can remain in place, but we would change the PaX Control configuration:

Since the PaX flags will now live on the extended attributions of your filesystems, you would need to enable xattr on those filesystems, but the PaX team has already set up a dependency. E.g. for Ext4 we have

Here  was automatically selected by. Nonetheless, in case you are using some exotic filesystem which doesn't have this this selection dependency, you may want to check and then file a bug report to have your filesystem better supported with respect to PaX.

PaX Control
As we mentioned above, there are five PaX protections that can be enforced (in SOFTMODE) or relaxed (in non-SOFTMODE) on a per ELF object basis: PAGEEXEC, EMULTRAP, MPROTECT, RANDMMAP and SEGMEXEC. The later, SEGMEXEC, is only available on x86 CPUs which support segmentation, unlike paging which is supported on all CPUs, even x86. Since some programs break for one reason or another under full PaX enforcement, we are faced with the choice of either fixing the code to work with PaX or relaxing one or more of these protections. In practice, "fixing the code" may be very difficult and we resort to the latter. The general approach should be to try full enforcement, and if something breaks, use dmesg to obtain a report from the kernel regarding why and then relax that particular protection. Even this is not generally needed on a Gentoo system because the ebuilds should set the correct flags for you via the pax-util.eclass. If you find that you have to set your own flags, we would ask that you file a bug report.

Generally setting the PaX flags is straightforward, but the user should keep a few things in mind:

1) One can set either PT_PAX and/or XATTR_PAX flags on the ELF object independently of one another. Similarly, the kernel can be configured to read either, both or neither fields. It is up to you to make sure that you set the flags in the field being used by the kernel to get the desired results. For example, if you have PT_PAX="Pe---" while XATTR_PAX missing on the object, but the kernel is configured only to use XATTR_PAX, you may not get the desired result!

2) The recommended approach is to mark both PT_PAX and XATTR_PAX fields identically on the objects whenever possible, and set the kernel to read only XATTR_PAX. The "whenever possible" is where things get complicated and the two fields may not wind up containing the same flags. If the ELF does not contain a PAX_FLAGS program header, PT_PAX marking will fail. However, the absence of this program header will not affect XATTR_PAX markings. If the ELF is busy (i.e. there is a running process making use of the ELF's text), then one can read the PT_PAX flags but not set them. Again, this does not affect setting or getting XATTR_PAX flags. On the other hand, if you are using any file systems which do not support extended attributes, then XATTR_PAX marking will fail on those file systems while PT_PAX marking is uneffected, except as already stated. This can be fairly subtle because copying a file from a file system with xattrs to one without, and then back again will drop the XATTR_PAX flags. Or tarring with an older version of tar which does not preserve xattrs will again drop our flags.

3) The PaX flags are only enforced when a process is loaded from an ELF executable. This executable in turn usually links dynamically against shared objects in memory. Using `cat /proc/ /status | grep PaX` gives you the resulting PaX enforcement on the running process with PID= . But since the executable and shared objects can have different flags, the question arises, which ones are used to determine the final running PaX enforcements? The answer is the executable for reasons of control and security. If the libraries were to set the runtime PaX enforcement, then which of the libraries would "win" if an executable linked against many? And one overly relaxed library could relax the privileges on many executables that link against it. E.g. Relaxing all PaX protection on glibc would effectively turn PaX off on one's system. Nonetheless, it may be the code in the library itself that needs the relaxation of some PaX enforcement. In that case, one has to "back port" the flags from the library to the executable that uses it.

Below we describe the utilities provided on a Gentoo system for working PaX markings. There are several since as PaX evolved, new features were needed. Each one emphasizes a different need with respect to the above caveats.

1. paxctl

This is the traditional upstream package for setting PaX flags. It is limited only in that it sets PT_PAX only, not XATTR_PAX. It is provided by emerging sys-apps/paxctl. It does have one functionality that no other utility has: it can either create a PAX_FLAGS program header or convert a GNU_STACK to PAX_FLAGS. Both of these are not recommended since they can break the ELF under certain circumstances. However, in the extreme case that you cannot use XATTR_PAX (e.g. you can't use file systems that support extended attributes) and you are dealing with an ELF object that wasn't build with a PAX_FLAGS program header, then these options are available to you via this utility.

Here is a synopsis of its usage:

Note that paxctl also reports on an older PaX protection called RANDEXEC. This is now deprecated --- the randomization of the base address of a processes now simply part of ASLR on all executables build ET_DYN rather than EX_EXEC. Here is paxctl in action:

2. getfattr setfattr

These are not PaX specific utilities but are general utilities to set a file's extended attributes. On Gentoo, they are provided by emerging sys-apps/attr. Since XATTR_PAX uses the user.* namespace, specifically it uses "user.pax.flags", you can use set/getfattr to work with this field. However, keep in mind that setfattr and getfattr know nothing about PaX, so they will not perform any sanity checking of what you are putting into that field. Only if you set user.pax.flags to some meaningful combination of the chars PpEeMmRr will the kernel respect your choice, else it falls back on the default. The following listing gives examples.

The following is an example of their usage for XATTR_PAX:

3. paxctl-ng

paxctl-ng is the new swiss army knife of working with PT_PAX an XATTR_PAX markings. It can be built with support for just one or the other or both types of markings. When built with support for both, it can copy PT_PAX to XATTRP_PAX fields or vice versa, to make sure you have consistency. In sum, it can do everything paxctl and set/getfattr can do, except it will not try to create or convert a PAX_FLAGS program header. This is discouraged and should only be use in the corner case mentioned above. Here is a synopsis of its usage:

paxctl-ng -h

The following is an example of paxctl-ng in action:

View both PT_PAX and XATTR_PAX fields (-v):

Set default '-' for PAGEEXEC (-Pp) for XATTR_FLAGS only (-l) and report the result (-v):

Delete the XATTR_PAX field altogether (-d) and report the result (-v):

Set "em" flags (-em) for XATTR_FLAGS only (-l) and report the result (-v):

Copy the XATTR_PAX to PT_PAX flags, overwriting the latter (-f) and report the result (-v):

Silently delete the XATTR_PAX field (-d but no -v):

View both PT_PAX and XATTR_PAX fields (-v):

4. Python module, pax.so, and a simple Python frontend, pypaxctl.

We also provide bindings to python via a module, pax.so, which is installed by emerging. This package is a dependency of since the both revdep-pax and migrate-pax import it. It can be compiled with either PT_PAX and/or XATTR_PAX support, like paxctl-ng, but it is not as featureful since its scope is limited to just the needs of revdep-pax and migrate-pax. This may change in the future if there is a need to better integrate PaX marking with portage which is written in python.

Currently pax.so publicly exports the following:

pypaxctl is a simple front end to pax.so which just gets and sets the PaX flags using the same logic. When getting the flags, if both PT_PAX and XATTR_PAX are present, the latter will override the former. When setting, it will set both fields whenever possible. Here it is in action:

Retrieve either PT_PAX or XATTR_PAX. The latter has priority if it exists. The canonical order is PEMRS.

It turns out that XATTR_PAX didn't exist, so we got PT_PAX.

Set some XATTR_PAX flags:

Now its reporting XATTR_PAX flags:

Set "me" on both PT_PAX and XATTR_PAX:

But what if we want PAGEEXEC off? Then set Pp for the default '-':

If the logic of XATTR_PAX taking precedence over PT_PAX is not what you want, it can be compiled with just PT_PAX xor XATTR_PAX support. In this case, the other field is not ever touched, as demonstrated below:

5. revdep-pax

This utility is used to map out the linkings between all the ELF objects on your system and their shared objects in both directions. It can then search for PaX flag markings that are mismatched between the shared objects than the executables; and optionally, allows the user to migrate the markings forwards or backwards on a per ELF object basis. Here's a synopsis of its usage:

Here's revdep-pax in action. Since the out is long, we've replaced some of it with ellipses:

Report all mismatching forward linkings:

Forward port PaX flags for python3.2 only:

Report all mismatching reverse linkings:

Let's migrate from the library's flags to the executable using the soname (-s). We could also have used /usr/lib64/libpython3.2.so.1.0 with the -l flag.

6. migrate-pax

At this point you're probably fed up with dealing with both PT_PAX and XATTR_PAX fields and their relationship to the kernel's configuration, and you just want to drop the older PT_PAX and get on with life! migrate-pax does only that ... it will go through all ELF objects on your system and migrate the PT_PAX field to XATTR_PAX. For more details on how to migrate, see our guide on migrating PaX flags from PT_PAX to XATTR_PAX.

Acknowledgements
We would like to thank the following authors and editors for their contributions to this guide:


 * Anthony G. Basile
 * Francisco Izquierdo
 * Brandon Hale
 * blackace
 * solar