Certificates

Certificates, also known as SSL certificates or TLS certificates, provide systems and users with a sound method for securely communicating with them or verifying those systems and users' identity. It contains identification information (who or what the certificate belongs to) and a public key, and is signed by a third party called the Certificate Authority.

This article will focus on dealing with certificates on Gentoo Linux. Where needed, a slight introduction will be given to the actions or sections presented. The technical details about certificates are better served on different platforms, such as Wikipedia.

Certificate authority
Certificate authorities (should) verify the identity of a user (or owner of a system) and, when verified, sign the certificate with their own private key. This signature is then used by others to assert that a presented certificate is valid and trustworthy. The idea here is that others trust the signature of a certificate authority, and through that trust also accept the presented certificate.

History has shown however that not all certificate authorities are equally trustworthy (take DigiNotar as a prime example). Also improvements in crypto-analysis show that some important parts (such as the hashing algorithm) can contain weaknesses. In 2008 for instance, it was proven that MD5 collisions can be used to create a rogue CA certificate. As such, keeping track of the trusted certificates and their current strength is advisable.

Whenever an application (including browsers) connect to systems over an SSL/TLS connection, part of the handshake is to verify if the presented certificate is signed by a trusted certificate authority. Hence, systems need to track which CA certificates are trustworthy. The collection of trusted CA certificates is often called a truststore.

Managing the truststores on a Linux system is a complex task as applications can use different approaches for handling their certificate-based operations. In the subsequent few sections the most common CA certificate handling approaches are documented.

OpenSSL-compatible ca-certificates
OpenSSL-compatible libraries use the directory as the (default) truststore. All trusted certificates are listed in this directory (generally through symbolic links to their actual location), together with a single file which has all the trusted CA certificates listed.

By default, this directory is automatically managed through the package, which uses the CA certificates from Mozilla's CA certificate store and repackages them for use with OpenSSL-based applications.

On most systems, this package will already be installed:

Upon installation, this package
 * 1) generates the  configuration file by listing all certificates inside  inside the configuration file
 * 2) calls the  script which reads in this configuration file and updates the  directory accordingly

The script also checks if any admin-provided certificates are in  and adds links to those certificates to the  directory as well.

Most users will be satisfied with the default certificate handling: it uses the CA certificate store from an upstream project that is actively pursuing the secure state of the certificates, and is automatically updated through Portage. However, administrators who want to have a more fine-grained control over which certificates are accepted for the truststore cannot update the file manually.

New installations of the will rewrite this configuration file, and the file is explicitly marked as not governed by Portage' configuration file protection:

NSS
Another popular library is Mozilla's Network Security Services or NSS. Unlike the OpenSSL-based certificate support, NSS uses database files as the certificate store.

NSS starts off with a hard-coded list of trusted CA certificates inside the file, installed through the  package. This list can be viewed from any of the applications using NSS capable of showing (and manipulating) the trust store, such as Chrome-compatible or Firefox-compatible browsers. However, another approach is to use the application as provided by the  package:

The trust attributes mentioned in the output show what trust level that attribute has and for which purpose. The man page has some information about what each attribute means. The most important ones are:


 * &mdash;Valid certificate authority
 * &mdash;Trusted certificate authority (implies )
 * &mdash;Valid peer certificate (i.e. no need to validate if it is signed by a proper CA)
 * &mdash;Trusted peer certificate (often used for self-signed certificates that need to be trusted)

The purpose of the trust attributes is for:


 * SSL&mdash;Secure communication over SSL/TLS channels
 * S/MIME&mdash;Encryption of messages and e-mail
 * JAR/XPI&mdash;Code signing

Alongside the hard-coded certificates, NSS supports a system-wide NSS databases located at, and user-specific NSS databases located at. The system-wide NSS database can contain the additional CA certificates that should be accepted by all users as well as system applications (and daemons) running on the system, while the user-specific database is - as the name implies - for specific user accounts.

To query the currently assigned CA certificates:

To list a certain certificate information in more detail, add the name:

Application-specific stores
Some applications have their own certificate store.

Mozilla Firefox
Although Firefox uses the NSS library for handling the secure communications, it does not use the location. Instead, it uses (alongside the hard-coded root CA certificates) NSS database files within its own profile, more specifically the file.

For instance, to list the intermediate CA certificates:

Raphaël Barrois describes, on his blog, how to get Firefox and Chrome to share the same location.

Adding trusted certificates
The most common operation against truststores is to add additional trusted certificates to it. Companies often have their own PKI (Public Key Infrastructure) setup which supports the internal certificate needs, and which might require the master certificate of this PKI to be trusted, or have it as an intermediate certificate deployed across the truststores in the organization. But even end users might want to add in additional trusted certificates who might not have made the "final" list yet as managed by Mozilla.

Obtaining the certificate
First the certificate itself needs to be obtained. When an organization has its own PKI then this CA certificate can be easily provided by the responsible team, with the necessary additional precautions in place to validate that the certificate is the right one. Other CAs generally distribute their CA certificates on their websites.

End users can also obtain the certificate through the browser by reviewing the SSL/TLS connection information. For instance, with Firefox, this can be obtained through the page information, Security section, View Certificate.

Whatever the source of the certificate, it is recommended to validate the certificate itself by checking (for instance) the checksum of the certificate. Gentoo's infrastructure uses certificates from DigiCert, who has its CA certificates published online together with the thumbprint information of the certificate.

For instance, the DigiCert High Assurance EV Root CA file has the following information published alongside the certificate itself:
 * serial number&mdash;a unique identifier for the certificate
 * thumbprint&mdash;an (in this case SHA1-based) checksum on the certificate

The downloaded file, which usually has a or  suffix, can be validated through the  command.

To view certificate information (including the serial number):

To view the thumbprint:

Including in the ca-certificates store
Once the certificate has been validated, store it in the location:

Run the command to update the  location with the new certificate.

Including in the NSS database
When applications use the NSS library and honor the location, then the following steps can be taken to include this certificate in this system-wide NSS database.

First, create the directory structure for the system-wide NSS database files:

Next, create a new database file set. A password will be asked to restrict editing of the database files to people who know the password. If all users on the system (and with access to the backups) are trustworthy, then this password can be kept empty.

Ensure that the database files are readable by everyone:

With the NSS database files now available, add the certificate to the store as follows:

The trust bits used in the above example mark the certificate as trustworthy for signing certificates used for SSL/TLS communication. The name used in the command is free to choose, but make sure to have it easily distinguished from other certificates in the store.

Similar instructions can be used to include the certificate only in the NSS database of a particular user:

Including in browser stores
Most browsers have an intuitive interface to handle the certificates they manage.
 * In Firefox, go to Preferences, Advanced, Certificates, View Certificates. In the Authorities tab, click on the Import button to open the dialog to import a certificate to the store.

Removing certificates
Certificates that were manually added using the instructions above can be removed from the stores easily as well.

Removing from ca-certificates store
To remove the certificate from the location, remove it from the  location and run the  command again.

Removing from NSS database files
To remove the certificate from any NSS database, use the command as follows. The example uses the system-wide NSS database location, but can be easily adjusted for the user-specific locations.

Modifying CA trusts
Although most administrators will be happy with the default CA handling, there will be situations where one wants to modify the default CA handling approaches. This could be due to - company policy requiring a fixed and company-approved list of CAs to be trusted, and nothing more - reports about trust issues with a root CA - the need or will to start off with an empty trust store and build up - requiring different trust stores per application

Whatever the reason, the next few sections will describe how to manipulate the previously discussed trust stores.

Modifying the OpenSSL-compatible ca-certificates
To handle custom CA lists, a secondary CA certificates configuration file needs to be managed, and after every installation of the package the  file needs to be overwritten and  needs to be executed again. This can be accomplished through Portage' sourcing.

Assuming that the secondary CA certificates configuration file is called. To have this file copied over to the file after every  package installation, the following script can be used:

All that the administrator has to do now is to keep the file up to date with the certificates of the CAs he wants to trust on this system. For instance, to only trust Verisign:

As was mentioned before, these entries are relative to the directory. Custom certificates can be added in and do not need to be listed in the  file for  to take them into account.

NSS-based modifications
The NSS library has three levels that might require any manipulations:
 * 1) the hard-coded list of trusted CAs
 * 2) the system-wide NSS databases
 * 3) the user- or application-specific NSS databases

Manipulating the hard-coded list
The recommended method for manipulating the hard-coded list is to change the trust attributes for the selected certificates in the system-wide NSS database. While the certificates are still listed in the file, they are then no longer marked as trusted.

Link in the hard-coded list, and then modify the trust bits for the certificate that needs modifying:

Modifying the NSS databases
Any manipulation to an NSS database file can be done through the command. More fancy user interfaces are available as well, but the commands are easy to use and to automate.

To remove a CA certificate from the store:

To add a root CA certificate to the store:

To add an intermediate CA certificate to the store:

To add a self-signed certificate to the store:

To modify the trust attributes of a certificate:

Debugging certificate issues
Certificate failures can be difficult to debug, not the least of which is due to the tools having hard-to-use interfaces. Here's some tips for debugging failures.

Formats
Certificates come in many different file formats, often use inconsistent or incorrect file extensions, and tools might expect inputs in specific formats. Try not to assume too much based purely on the file extension. Start with a tool like  or   to see if you can figure out what format it's in.

If you have a DER format, that's a binary format meant only for tools to read. You can usually convert it to a PEM format that contains the same information, but as plain text. Some tools only accept PEM, so try converting it:

Once it's the PEM format, you can easily view it:

Fetching certificates from network services
If you want to see what certificates a site is presenting over the network, use OpenSSL to start a connection and dump various useful details:

To save the certificates for further analysis, simply redirect the output, then use your typical pager.

Checking the certificate chain
OpenSSL's  command will display the certificate chain. Look at example.com:

The first certificate [subject]  is the example.com server. It's signed by the intermediate certificate [issuer]  which is signed by the root certificate. The root certificate then signs itself, and while self-signed certificates are normally not accepted, since it's in the truststore, it's trusted to do that.

OpenSSL's  command can verify certificates manually, but it too only operates on the first certificate found in the file. Extract the individual certificates out and double check they extracted correctly:

Finally the certificates can be verified. Walk back from the root certificate which should be accepted because it's in the truststore:

Then the intermediate certificate should be accepted because its issuer is in the truststore:

Then the server certificate should fail! This is because the verify command only checks the truststore, and while the root certificate is in there, the intermediate certificate is not.

To verify the server certificate, provide the intermediate certificate at the same time. OpenSSL will take care of walking the chain of trust. Be sure to use  so the certificate isn't inadvertently added to the truststore.

How are new CA's added to Gentoo?
Gentoo does not accept any custom certificate authorities. The project does not have the time, resources, or expertise to properly vet certificate authorities which is why we utilize NSS for its truststore.

If you want a new root CA added to the default Gentoo install, please go through Mozilla's CA Certificate Program. Once it's accepted by Mozilla, updating Gentoo is pretty easy.

My certificate used to be recognized, but no longer is!
If your certificate used to work but no longer does, the root certificate might have been removed from the truststore because it had problems (e.g. was insecure). See Mozilla's list of removed CA's for more information.

This could also explain why a website used to work but is now marked as insecure.

Why is a certificate not recognized and/or marked as insecure?
All certificates must be trusted by an entry in the truststore, either directly by a root certificate in the truststore (which is possible, but a bit uncommon), or indirectly by intermediate certificates that themselves chain back to a trusted root certificate. The Wikipedia Chain of trust article goes into a lot more detail, as does Let’s Encrypt Chain of Trust article.

Usually when a certificate is unable to be verified, it's because an intermediate certificate is missing. For example, a service presented the last certificate in the chain (its own) and expected you to either have the intermediate certificate already, or be able to fetch it from somewhere. Both of these assumptions are usually incorrect and the service should actually be providing a certificate bundle instead: not only their own certificate, but the intermediate certificates back to the relevant root certificate that is in the truststore. This way the client is able to follow the trust chain back to its own truststore.

If your certificate chain ends in a root certificate that is not in Gentoo's default truststore, see the previous FAQs.

Can intermediate certificates be added to the truststore?
See the previous FAQ about adding a new CA to Gentoo to start with. Building off of that answer, intermediate certificates are typically not included in the truststore. Maintaining the existing list of root certificates requires a lot of effort & toil and there's only <150 of those. There are orders of magnitude more intermediate certificates out there (thousands, if not more) that individual certificate authorities create & destroy as makes sense for their needs & structure. Requiring review by Mozilla, and then updates rolling out to distros & users, adds a large amount of toil for no gain.

This is the status quo for all operating systems, browsers, distros, and system builders. Gentoo alone including an arbitrary intermediate certificate won't actually help any users on all those other systems, and servers would still have to provide them to keep everyone working.

External resources

 * Certificate Authorities (CA) in Google Chrome / Chromium and Firefox on Linux on Roland's Blog
 * Linux Cert Management in Chromium's developer documentation
 * Certificate Authority FAQ on the Mozilla wiki
 * Comparison of SSL/TSL libraries that libcurl can be built to use.