Daemontools-encore

Daemontools-encore is a backwards compatible, enhanced version of Daniel J. Bernstein's daemontools package, written by Bruce Guenter. A summary of the features that have been added to daemontools-encore is available here.

Environment variables

 * SUPERVISEDIR - Name of the the subdirectory used for 's control files in a service directory.
 * SOFTLIMIT_ALLBYTES - Alternative to 's -a option for specifying the corresponding soft limit.
 * SOFTLIMIT_COREBYTES - Alternative to 's -c option for specifying the corresponding soft limit.
 * SOFTLIMIT_DATABYTES - Alternative to 's -d option for specifying the corresponding soft limit.
 * SOFTLIMIT_FILEBYTES - Alternative to 's -f option for specifying the corresponding soft limit.
 * SOFTLIMIT_LOCKEDBYTES - Alternative to 's -l option for specifying the corresponding soft limit.
 * SOFTLIMIT_MEMBYTES - Alternative to 's -m option for specifying the corresponding soft limits.
 * SOFTLIMIT_OPENFILES - Alternative to 's -o option for specifying the corresponding soft limit.
 * SOFTLIMIT_PROCS - Alternative to 's -p option for specifying the corresponding soft limit.
 * SOFTLIMIT_RSSBYTES - Alternative to 's -r option for specifying the corresponding soft limit.
 * SOFTLIMIT_STACKBYTES - Alternative to 's -s option for specifying the corresponding soft limit.
 * SOFTLIMIT_CPUSECS - Alternative to 's -t option for specifying the corresponding soft limit.

Files

 * - Location of the scan directory when using OpenRC, or.

OpenRC
See here

Usage
Bernstein daemontools and daemontools-encore implement process supervision: programs that run as long-lived processes, such as a server program, can be supervised by being run as a child process of a supervisor. The supervisor can detect if the process, also called the service or the daemon in this context, has unexpectedly stopped running, e.g. because it exited with an error status or was killed by a signal, and automatically restart it. The supervisor also provides a reliable interface for controlling both the supervised process and itself, to send signals to the process, and to query status information about it.

All this is based on standard POSIX features: a process' parent reliably knows its child's process ID (PID), because it is the return value of the call that creates it, knows when the child stops running, because it is notified with a   signal, and can obtain exit status information when it happens using the  or  calls.

For more information information about process supervision see.

Service directories
The program implementing the supervisor features in Bernstein daemontools and daemontools-encore is. Supervision for a single process is configured using a service directory (or servicedir). A servicedir is an ordinary directory containing at least one executable file named. It can also contain an optional, regular file named. The (absolute or relative to the working directory) pathname of this servicedir is then passed as an argument to. This however is not supposed to be done directly by the user, but to happen indirectly as a consequence of running.

When is invoked, it changes its working directory to the specifed servicedir, and executes the contained  file as a child process, unless there is also a  file. Daemontools-encore's also makes the child process the leader of a new session using the POSIX  call, unless the servicedir contains a regular file named. In that case, the child process will run in 's session instead. Making the child process a session leader with Bernstein daemontools requires using the program inside  (see supervised process state changes). If is invoked with a servicedir that contains a  file, the  file won't be executed, but the service can be started later with the  program (see controlling supervised processes). The contents of the and  files are ignored, so they are usually empty.

can have any file format that the kernel knows how to execute, but is usually a shell script that performs some sort of initialization, and then calls the real program intended to be supervised, using the shell's builtin utility. This allows the program to run without creating a new process, so it will have the same PID as the script, and from there on become the supervised process. If the supervised process exits, will wait 1 second before restarting it, so that it does not loop too quickly.

Programs that fail to adhere to certain design criteria, including those that use in order to "put the daemon into the background", might not be able to be supervised. Sometimes programs can meet those criteria if passed certain options (e.g. a 'run in the foreground' option) as arguments on invocation.

A minimal (and simplistic) service directory for a hypothetical program:

The program keeps control files in a subdirectory of the servicedir, also named. If this subdirectory or any of its files doesn't exist when  is invoked, they will be created. If the servicedir contains a symbolic link to directory instead of a subdirectory,  will follow it and use the linked-to directory for its control files.

Service directories can also optionally contain a subdirectory or symbolic link to directory named, which must also have the format of a servicedir, and can be used for setting up a dedicated logger (see logging).

The daemontools-encore package also allows the service directory to optionally contain executable files named, , and. These can be used to perfom initialization and cleanup actions, either first-time only, last-time only, or each time the supervised process starts or stops, and to conditionally perform actions based on its exit status information. It also allows to be optional, allows  to be an executable file instead of a directory, can set a different name for the directory containing 's control files via the SUPERVISEDIR environment variable, and avoids restarting a supervised process if its exit code is 100. For further details, please consult the man page.

Service directory with a subdirectory:

subdirectory contents:

The scan directory and supervision tree
Bernstein daemontools and daemontools-encore allow supervising a set of processes running in paralell using the program and a scan directory (or scandir). A scan directory is an ordinary directory containing subdirectories and/or symbolic links to directories, each of them a service directory. Invoking with the (absolute or relative to the working directory) path of the scandir as its first argument (and only argument for Bernstein daemontools' ) launches one child process for each subdirectory, unless the name of the directory starts with a dot. If is called with no arguments, it will assume the working directory is the scandir, otherwise it will change its working directory to the specified scandir.

Every 5 seconds, rereads the scan directory (i. e. performs a scan), launching  processes for each new servicedir it finds, or old servicedir for which it finds its  process has exited. acts as a supervisor for every child, the same way does for its child process. This arrangement of processes is called the supervision tree. The root of the supervision tree is, so it is designed to be robust and to cope with processes exiting unexpectedly, being killed, or being unable to start. The leaves of the supervision tree are the processes that correspond to the service directories in the scandir.

Example scan directory containing three service directories:

Resulting supervision tree when is run on this scandir as a background process in an interactive shell, assuming it is a subdirectory named  in the working directory (i.e. launched with svscan scan & ):

Services directories using and daemontools-encore,  and  files:

Resulting supervision tree:

Note that since has a  file, the corresponding  process has no children.

Messages printed to svscan's standard output (stdout) by the script:

For further details, please consult the man page.

The multilog program
If a servicedir S in the scan directory contains a subdirectory or symbolic link to directory,  will launch two  processes in paralell, one executing  as a child process, and the other executing  with its standard input (stdin) connected to 's standard output (stdout) by a pipe. If any of the two processes or their  parents stops running and is restarted, the same pipe is reused so that no data is lost. This allows per-service logging by having execute a logger program. Bernstein daemontools and daemontools-encore packages provide such a logger: the program. This works for programs that send messages to their standard error (stderr).

is invoked as multilog arg1 arg2 arg3 ..., and expects to read a sequence of newline-terminated lines of text from its standard input. The arguments arg1, arg2, arg3, ... make up a logging script that tells what to do with them. Each argument specifies an action, actions are carried out sequentially in argument order.

The simplest form of invocation is multilog dir, where dir starts with a dot ('.') or a slash ('/'): it is interpreted as a pathname specifying an automatically rotated logging directory (or logdir). The logdir contains control files used by, a current log file named , and may also contain a set of old log files. The current log file contain a selection of the lines read by, possibly modified by other actions specified in its logging script, and old log files are produced by rotations: when approaches a certain maximum size, its contents are copied to a another file, after some optional processing, and  is emptied. Old log file have names beginning with '@', continuing with a timestamp in external TAI64N format showing when the file was finished, and ending with either '.s' or '.u'. The .s files are files that have been completely processed and safely written, i.e. files produced by a complete rotation. The .u files, if any, are files created by an incomplete rotation, they are not completely processed and may be truncated. Rotations respect line boundaries, i.e. old log files will always contain whole lines. Also, to avoid indefinite accumulation of old log files, if, after completing a rotation, their number exceeds a certain value, deletes the oldest log file (the file with the smallest TAI64N timestamp). The default maximum size of a file is 99999 bytes, and the default maximum number of old log files is 10. A rotatin can be forced by sending a   signal. If it is running as a supervised process, the program can be used to do it (see controlling supervised processes).

Some other actions recognized by in a logging script are:


 * + (plus sign) and - (minus sign) followed by a pattern: By default, logs all the lines read from its standard input without modifications; these actions allow selection and deselection of lines for logging, respectively, based on the specified pattern. Because all lines are initially selected, + actions are only effective after a - action deselecting some lines first. The pattern uses the shell's notation for matching strings, and must match whole lines (i.e. partial matching doesn't count), with the following restrictions:
 * Berstein daemontools' only treats asterisks ('*') as special characters in the pattern, they match any string, including the null string, that does not include the next character in pattern (instead of the POSIX behaviour of matching the greatest possible number of characters that still allows the remainder of the pattern to match the line), and doesn't support escaping them with a backslash ('\').
 * Daemontools-encore's can switch between Bernstein daemontools behaviour and using the full range of patterns allowed by POSIX, including backslash escaping. The latter is more flexible at the expense of being less efficient. At the start of logging script action processing, Bernstein daemontools match mode is selected.
 * An F action selects match mode for all subsequent + and - actions until the next mode-changing action in the logging script.
 * An S action selects Bernstein daemontools match mode for all subsequent + and - actions until the next mode-changing action in the logging script.
 * t: prepends each logged line with '@', followed by a timestamp in external TAI64N format (printed as 24 lowercase hexadecimal characters), and a space. It must be specified as the first action, and any subsequent + and - actions match against the line with the prepended timesptamp. Daemontools-encore also supports a T action, which must also be the first one, that prepends an accustamp-style timestamp and a space instead ( was a program shipped with Bernstein daemontools that was eventually dropped).
 * s followed by an integer: sets the maximum size of the file to the specified value (in bytes), for all subsequent actions in the logging script specifying a logdir, until the next s action.
 * u followed by an integer: sets the maximum number of old log files to the specified value, for all subsequent actions in the logging script specifying a logdir, until the next u action.
 * ! (exclamation mark) followed by a program name (that must be capable of finding using PATH search): allows  to perform some processing of the contents of the  log file during a rotation.  will call the program and feed it the contents of  on its standard input. The .s or .u file produced as a result will contain that program's output.

For details about all actions supported, and about the protocol used for communication between and the processor specified in an ! action, please consult the  man page.

When there's no process running on a logdir, the  file has the executable by user flag set.

Note that the second log file also contains the warning messages. This shows that actions that aren't + or - do not affect the currently selected lines, so warning messages selected by +warning: * stay selected after processing the ./log1 action. To only have the error messages in the log file, a -* action should have immediatly preceded the +error: * action.

Service directory containing a logger:

Note that since makes the service directory its working directory, the relative pathname in 's invocation results in the logging directory being placed wherever the scan directory is.

Resulting supervision tree:

Contents of the logging directory:

This logdir contains two old log files and, since is running, the  file has its executable by user flag cleared.

The TAI64N timestamps can be shown in human-readable form with, for further details please consult the respective man page.

The readproctitle program
The program expects to read a sequence of newline-terminated lines of text from its standard input, and saves them to an automatically rotated log it keeps in memory. The log shows up in the output of the utility. The number of characters displayed is specified by 's arguments.

is invoked as readproctitle arg1 arg2 ... argn dots. The arguments arg1, arg2, ..., argn can be anything, they are shown as-is in the output of, and can used for displaying some kind of heading that introduces the log. The last argument, dots, must be at least five dots ('.'), and its length specifies the number of characters the log kept in memory will have. In the output of, 's last shown argument will be the contents of the log up to that time instead. Characters shift to the left as reads further lines of text, and older characters are discarded to keep the log of constant size. Therefore, the command will show each time the most recent data. For further details, please consult the man page.

After enough seconds have elapsed:

The best known use of is in the  script included in Bernstein daemontools and daemontools-encore.

The logging chain
A supervision tree where all leaf processes have a logger can be arranged into what the author of s6 calls the logging chain, which he considers to be technically superior to the traditional syslog-based centralized approach.

Since processes in a supervision tree are created using the POSIX call, all of them will inherit 's standard input, output and error. A logging chain arrangement using Bernstein daemontools and daemontools-encore is as follows:


 * Leaf processes should normally have a logger, so their standard output and error connect to their logger's standard input. Therefore, all their messages are collected and stored in dedicated, per-service logs by their logger. Some programs might need to be invoked with certain options passed as arguments to make them send messages to their standard error, and redirection of stderr to stdout (i.e. 2>&1 in a shell script) must be performed in the servicedir's file.
 * Leaf processes with a controlling terminal are an exception: their standard input, output and error connect to the terminal.
 * , the loggers, and leaf processes that exceptionally don't have logger for some reason, inherit their standard input, output and error from, so their messages are sent wherever the ones from are.
 * Leaf processes that still unavoidably report their messages using have them collected and logged by a (possibly supervised) syslog server.

Specifying a logger for
Daemontools-encore's allows the pathname of a service directory to be passed as a second argument after the scan directory's pathname. If this argument is present, will launch one  process for the specified directory, and connect its standard output and error to the corresponding supervised process' standard input by a pipe. This makes it possible to set up a logger for in the same way a logger can be set up for the leaf processes of the supervision tree.

Supervised process state changes
Bernstein's daemontools and daemontools-encore provide a set of tools for modifying a supervised process' execution state. These tools employ a technique called chain loading by some people, and Bernstein chaining by others. A program prog1 designed to use chain loading is invoked as prog1 arg11 ... arg1n prog2 arg21 ... arg2n, where prog2 is the name of another program. When prog1 is invoked, it performs some action based on arguments arg11, ..., arg1n, and then executes prog2 without creating a new process, using one of the POSIX calls. Arguments arg21, ..., arg2n are not interpreted by prog1 and are passed along to prog2.

If prog2 is also designed to use chain loading, some of those arguments might in turn be the name of a third program prog3 and its corresponding arguments, which will be executed after prog2 completes its task. As a consequence, it is possible to build a chain of programs that will all run in the same process, therefore preserving the process ID (PID), which makes the technique suitable for the script of a service directory. The final one in such a chain of programs would be the real program intended to be supervised.

The tools provided by Bernstein's daemontools and daemontools-encore to change a process' execution state are:


 * : runs another program with environment modified according to files in a specified directory.
 * : runs another program with environment variables indicating a specified account's user ID and group ID
 * : runs a program in a new session (using POSIX ).
 * : runs another program with a file locked (using Linux on Gentoo).
 * : runs another program under a specified account's user ID and group ID. Requires root privileges.
 * : runs another program with new soft resource limits (using POSIX ). These limits are set via options specified as 's arguments. Daemontools-encore also allows setting those limits via SOFTLIMIT_* environment variables, applied at invocation.

For further details on these tools please consult their respective man page.

The directory supplied to the program is called an environment directory (or envdir). Each file in the envdir controls a single environment variable. If the file is empty, and a variable with the same name as the file exists in the process' environment, it is removed from it. If the file is nonempty, the contents of its first line become the value of a new environment variable with the same name as the file. If the variable existed before 's invocation, it is first removed from the environment, and then readded with the new value. Enviroment variables not referred to by a file in the envdir remain unchanged.

This script allows executing program with effective user daemon and the maximum number of open file descriptors set to 5. This is the same as if performed a  call itself with rl.rlim_cur set to 5, provided that value does not exceed the corresponding hard limit. As in previous examples, the redirection of stderr to stdout allows setting up a dedicated logger for.

Example service directory with an enviroment directory :

This script adds variables UID, GID and SOFTLIMIT_OPENFILES to the environment, the first two set to the user ID and group ID of user daemon via , and the last one via the enviroment directory , which is used by the invocation to set the maximum number of open file descriptors, provided it is the daemontools-encore version of that program. It is assumed that can use environment variables UID and GID to drop privileges.

Controlling supervised processes
Bernstein daemontools both and daemontools-encore provide two programs for controlling a supervised process and to query status information about it: and, respectively. Daemontools-encore provides a third one, ; for further details please consult its man page.

The program accepts a set of service directory pathnames and options that specify what to do. Some of those options are:


 * svc -d (down): If the supervised process is running, send it a  signal followed by a   signal. The  file won't be reexecuted after that. This is the standard way to manually stop a supervised process. If the process is not running (e.g. because it has a  file or after a previous svc -d command) this is a no operation.
 * svc -u (up): If the supervised process is not running, start it by executing its file. If the process is running, this is mostly a no operation, except that it cancels the effect of a previous svc -o command. This is the standard way to manually start a supervised process that has a  file.
 * svc -o (once): If the supervised process is not running, start it by executing its file, but don't restart it if stops. This still allows the process to be monitored and to be reliably sent signals, but it won't actually be supervised. If the process is running, it just asks  to not restart it if it stops. The effect of this option can be cancelled by a subsequent svc -u command, and restored after that by another svc -o command.
 * svc -x (exit): Asks to exit if the supervised process stops. This is only effective if its  parent stopped running without  being killed, otherwise a new  process will be launched on the next periodic scan.

Other options allow reliably sending signals to a supervised process. For further details, please consult the man page. In particular, svc -a can be used to send a  signal to a supervised  process to force it to perform a rotation.

The program accepts a set of service directory pathnames. Its output is intended to be displayed on a user interface or logged, not to be fed as input to a program for parsing. Bernstein daemontools' displays whether the supervised process is running ('up') or not ('down') and whether it is transitioning to the desired state or already there ('want up' or 'want down'), its process ID (PID) if it is up, and whether its current up or down status matches the presence or absence of a  file in the servicedir ('normally up' or 'normally down'). If also shows if the supervised process is paused (because of a  signal). In addition to that, daemontools-encore's displays an extended state, 'started', 'starting', 'running', 'failed', 'stopping' and 'stopped', that depends on which of the,  or  files in the servicedir (if present) is being executed, and the reason it stopped running, if it did. It can also display similar information for the logger, depending on options passed as arguments, if there is a executable file or directory in the servicedir. For further details, please consult the man page.

Starting the supervision tree
In a supervision tree arrangement each process supervises its children, therefore the robustness of the supervision tree ultimately relies on the process. So the problem of how to supervise itself arises, as well as what to do with 's messages in a logging chain

From OpenRC
Gentoo's packaging of Bernstein daemontools and daemontools-encore provides an OpenRC service script named, to run. The scan directory will be. Thus, is can be started when the machine boots by adding it to an OpenRC runlevel using :

Or it can also be started manually:

The svscanboot script
Bernstein daemontools and daemontools-encore provide a script called, that can be launched and supervised by sysvinit by adding a 'respawn' line for it in. is a shell script that launches an process and a  process, with the former's standard output connected to the latter's standard input by a pipe, and every other standard descriptor redirected to. The enviroment will be empty, except for the PATH variable, set to a known value in the script. The scan directory will be.

Used in this way, the supervision tree becomes rooted in process 1, which cannot die without crashing the machine. Also, 's messages will go to, so they can be seen using the utility.

Gentoo users wanting to use will need to manually edit, and then call

This will make sysvinit launch and supervise when entering runlevels 1 to 5. Because calls  using absolute path, a symlink to the correct path for Gentoo must be created:

The supervise-scripts package
Bruce Guenter's supervise-scripts package provides an script that also allows sysvinit to launch and supervise. The script will make a backup of (with a name of the form ) and then modify it so that another script,  is launched and supervised when entering runlevels 1 to 5. The scan directory will be.

The script is a wrapper around. After some initialization it will replace itself with using the shell's  builtin utility, with its standard input, output and error redirected to. Therefore, 's messages will be shown on the console. The enviroment will be empty, except for the PATH variable, set to a known value in the script.

The package also includes a script, that will be called when entering sysvinit's single user mode or runlevels 0, 1 and 6, to perform cleanup. It will stop all lingering processes and their children using an svc -dx command. The script makes use of, so it can only be used with daemontools-encore.

Unmerge
All scan directories, service directories, the symlink, etc. must be manually deleted if no longer wanted after unmerging the package. Also, all modifications to must be manually reverted, even when using the supervise-scripts package, because  does not have an 'undo' option. Lines for, and  must be deleted, and a telinit q command must be used afterwards. If no further changes to have been made after invoking, the backup copy created by that script before the changes are applied can be used to revert them by just replacing  with that file.

External Resources

 * Tokiclover's supervision framework (OpenRC friendly)
 * Avery Payne's supervision-scripts project