Daemontools-encore

Daemontools-encore is Daniel J. Bernstein's daemontools package, written by Bruce Guenter]]. A summary of the features that have been added to daemontools-encore is available here.

Environment variables

 * SUPERVISEDIR - Name of the subdirectory of a service directory used by for its control files. If it is an absolute pathname, it will be suffixed by all daemontools-encore programs that use it with the path to the service directory with all slashes ('/') replaced with colons (':'), e.g. if SUPERVISEDIR is set to, a  process running for service directory  will place its control files in directory.
 * SOFTLIMIT_ALLBYTES - Alternative to 's  option for specifying the corresponding soft limit.
 * SOFTLIMIT_COREBYTES - Alternative to 's  option for specifying the corresponding soft limit.
 * SOFTLIMIT_DATABYTES - Alternative to 's  option for specifying the corresponding soft limit.
 * SOFTLIMIT_FILEBYTES - Alternative to 's  option for specifying the corresponding soft limit.
 * SOFTLIMIT_LOCKEDBYTES - Alternative to 's  option for specifying the corresponding soft limit.
 * SOFTLIMIT_MEMBYTES - Alternative to 's  option for specifying the corresponding soft limits.
 * SOFTLIMIT_OPENFILES - Alternative to 's  option for specifying the corresponding soft limit.
 * SOFTLIMIT_PROCS - Alternative to 's  option for specifying the corresponding soft limit.
 * SOFTLIMIT_RSSBYTES - Alternative to 's  option for specifying the corresponding soft limit.
 * SOFTLIMIT_STACKBYTES - Alternative to 's  option for specifying the corresponding soft limit.
 * SOFTLIMIT_CPUSECS - Alternative to 's  option for specifying the corresponding soft limit.

Files

 * - Location of the scan directory when using OpenRC, or.

OpenRC
See here

Usage
Bernstein daemontools and daemontools-encore implement process supervision: programs that run as long-lived processes, such as a server program, can be supervised by being run as a child process of a supervisor. The supervisor can detect if the process, also called the service or the daemon in this context, has unexpectedly stopped running, e.g. because it exited with an error status or was killed by a signal, and automatically restart it. The supervisor also provides a reliable interface for controlling both the supervised process and itself, to send signals to the process, and to query status information about it.

All this is based on standard POSIX features: a process reliably knows its child's process ID (PID), because it is the return value of the  call that creates it, knows when its child stops running, because it is notified with a   signal, and can obtain exit status information when it happens using the   or   calls.

For more information information about process supervision see.

Service directories
The program implementing the supervisor features in Bernstein daemontools and daemontools-encore is. Supervision for a single process is configured using a service directory (or servicedir). A servicedir is an ordinary directory containing at least one executable file named. It can also contain an optional, regular file named. The (absolute or relative to the working directory) pathname of this servicedir is then passed as an argument to. This however is not supposed to be done directly by the user, but to happen indirectly as a consequence of running.

When is invoked, it changes its working directory to the specifed servicedir, and executes the contained  file as a child process, unless there is also a  file, or, for daemontools-encore only, a  file (see the,  and  files, and the daemontools-encore extended service state). Daemontools-encore's also makes the child process the leader of a new session using the POSIX   call, unless the servicedir contains a regular file named. In that case, the child process will run in 's session instead. Making the child process a session leader with Bernstein daemontools requires using the program inside  (see supervised process execution state changes). If is invoked with a servicedir that contains a  file, the  file won't be executed, but the service can be started later with the  program (see controlling supervised processes). The contents of the and  files are ignored, so they are usually empty.

can have any file format that the kernel knows how to execute, but is usually a shell script that performs some sort of initialization, and then calls the real program intended to be supervised, using the shell's builtin utility. This allows the program to run without creating a new process, so it will have the same PID as the script, and from there on become the supervised process. waits for 1 second between two child process spawns, so that it does not loop too quickly if the process exits immediately. The daemontools-encore version of also has special behaviour when it receives a signal: if it receives a   signal, it behaves as if an svc -dx command naming the corresponding servicedir had been used (see controlling supervised processes), if it receives a   signal, it sends a   signal to the supervised process, as if an svc -p command naming the corresponding servicedir had been used, and if it receives a   signal, it sends a   signal to the supervised process, as if an svc -c command naming the corresponding servicedir had been used.

Programs that fail to adhere to certain design criteria, including those that use  in order to "put the daemon into the background", might not be able to be supervised. Sometimes programs can meet those criteria if passed certain options (e.g. a 'run in the foreground' option) on invocation.

A minimal (and simplistic) service directory for a hypothetical program:

The program keeps control files in a subdirectory of the servicedir, also named. If this subdirectory or any of its files doesn't exist when  is invoked, they will be created. If the servicedir contains a symbolic link to directory instead of a subdirectory,  will follow it and use the linked-to directory for its control files. Daemontools-encore also allows setting the name of this control directory via the SUPERVISEDIR environment variable, see environment variables.

Service directories can also optionally contain a subdirectory or symbolic link to directory named, which must also have the format of a servicedir, and can be used for setting up a dedicated logger (see logging).

For further details about the program and service directories, please consult the  man page.

Service directory with a subdirectory:

subdirectory contents:

The scan directory and supervision tree
Bernstein daemontools and daemontools-encore allow supervising a set of processes running in parallel using the program and a scan directory (or scandir). A scan directory is an ordinary directory containing subdirectories and/or symbolic links to directories, each of them having the structure of a service directory. Invoking with the (absolute or relative to the working directory) path of the scandir as its first argument (and only argument for Bernstein daemontools' ) launches one child  process for each subdirectory, unless the name of the directory starts with a dot ('.'). If is called with no arguments, it assumes the working directory is the scandir, otherwise it changes its working directory to the specified scandir.

Every 5 seconds, rereads the scan directory (i. e. performs a scan), launching  processes for each new servicedir it finds, or old servicedir for which it finds its  process has exited. acts as a supervisor for every child, the same way does for its child process. This arrangement of processes is called the supervision tree. The root of the supervision tree is, so it is designed to be robust and to cope with processes exiting unexpectedly, being killed, or being unable to start. The leaves of the supervision tree are the processes that correspond to the service directories in the scandir.

Example scan directory containing three service directories:

Resulting supervision tree when is run on this scandir as a background process in an interactive shell, assuming it is the working directory (i.e. launched with svscan & ):

The, and  files, and the daemontools-encore extended service state
Daemontools-encore service directories can contain executable files named, and. The file is optional for daemontools-encore, but either  or  must exist in the servicedir. If there is a file and it is executable, it will be executed as a child process instead of  when  is invoked, and also when an svc -u or svc -o command is used to manually start the service (see controlling supervised processes). If the process exits with a an exit code of 0,  will then execute the  file just like Bernstein daemontools'. can be used to perform some kind of first time-only initialization for the program intended to be supervised.

If the supervised process exits with an exit code of 100, or if the process is killed by a signal or exits with a nonzero exit code,  enters the failed extended state (see later) and does not restart the service. If the service is manually stopped with an svc -d command or, after an svc -o command is used, the supervised or process stop running and the conditions for entering the failed state are not met,  enters the stopped extended state. In stopped state, if there is a file in the servicedir and it is executable,  will execute it as a child process. can be used to perform some kind of last time-only cleanup for the program intended to be supervised, or for the process. If the process exits with an exit code of 100,  will also enter the failed state, otherwise it will remain in stopped state.

If there is a file,  will execute it as a child process each time a,  or  process is spawned or stops running for whatever reason, so it can be used to perfom some action after one or more of those events. invokes with four arguments: the first one is either start, run or stop, indicating which process caused  to be invoked, the second one is either start, exit or killed, indicating that the corresponding process was spawned, exited or was killed by a signal, respectively, the third one is the process ID of the corresponding process, and the fourth one is the exit code or signal number, as determined by POSIX , or 0 if the second argument is start. So, for example, could be used to perfom cleanup actions each time the supervised process stops, possibly depending on its exit status information, by checking that its first argument is run and its second argument is other than start, behaving like a runit or s6  file.

The effect of the file applies to all, ,  and  processes.

Because Bernstein daemontools' only executes a  file, the corresponding service can only be in two states: up if  has a child process, or down if it doesn't. Daemontools-encore's, on the other hand, can have at any given time either a , ,  or  child process, so for compatibility with Bernstein daemontools, when the service state is queried with the  program (see controlling supervised processes), an up or down state will still be displayed, but also an additional extended state: starting, started, running, stopping, stopped or failed.


 * Extended state starting will be displayed after launches a  process.
 * Extended state started will be displayed after the process exits successfully if there is no  file. In started state,  does nothing but wait for commands sent with the  program.
 * Extended state running will be displayed after launches a  process.
 * Extended state stopping will be displayed after sends its child process a   signal followed by a   signal to make it stop, until the process dies.
 * Otherwise, either extended state stopped or extended state failed will be displayed. In failed state, or in stopped state after the process has exited,  does nothing but wait for commands sent with the  program.

Whether a process is currently running or not does not modify the service state, it will be whatever it was immediatly before  was executed.

Services directories using, , and  files:

Resulting supervision tree:

Note that since has a  file, the corresponding  process has no children.

Messages printed to 's standard output (stdout) by the script:

Sending a  signal to the  process (PID 2287):

Manually stopping service test-service2:

The multilog program
If a servicedir S in the scan directory contains a subdirectory or symbolic link to directory named, will launch two  processes in parallel, one executing S as a child process, and the other executing S with its standard input (stdin) connected to S's standard output (stdout) by a pipe. If any of the two processes or their parents stops running and is restarted, the same pipe is reused so that no data is lost. This allows per-service logging by having S execute a logger program. Bernstein daemontools and daemontools-encore packages provide such a logger: the program. This type of logging works for programs that send messages to their standard error (stderr).

Daemontools-encore's allows  to be an executable file instead of subdirectory or symlink to directory. In this case, the same process will supervise both the 'main' service and the logger, and maintain the pipe between then. This is similar to having a subdirectory with a file equivalent to the  file, and no  or  files. The file of the 'main' service will be invoked with log as its first argument when the  process starts, exits or gets killed by a signal. The and  files of the 'main' service also apply to the  process in this case.

is invoked as multilog arg1 arg2 arg3 ..., and expects to read a sequence of newline-terminated lines of text from its standard input. The arguments arg1, arg2, arg3, ... make up a logging script that tells what to do with them. Each argument specifies an action, actions are carried out sequentially in argument order.

The simplest form of invocation is multilog dir, where dir starts with a dot ('.') or a slash ('/'): it is interpreted as a pathname specifying an automatically rotated logging directory (or logdir). The logdir contains control files used by, a current log file named , and may also contain a set of old log files. The current log file contains a selection of the lines read by, possibly modified by other actions specified in the logging script, and old log files are produced by rotations: when approaches a certain maximum size, its contents are copied to a another file, after some optional processing, and  is emptied. Old log file have names beginning with '@', continuing with a timestamp in external TAI64N format showing when the file was finished, and ending with either .s or .u. The .s files are files that have been completely processed and safely written, i.e. files produced by a complete rotation. The .u files, if any, are files created by an incomplete rotation, they are not completely processed and may be truncated. Rotations respect line boundaries, i.e. old log files will always contain whole lines. Also, to avoid indefinite accumulation of old log files, if, after completing a rotation, their number exceeds a certain value, deletes the oldest log file (the file with the smallest TAI64N timestamp). The default maximum size of a file is 99999 bytes, and the default maximum number of old log files is 10. A rotation can be forced by sending a   signal. If it is running as a supervised process, the program can be used to do that (see controlling supervised processes).

Some other actions recognized by in a logging script are:


 * + (plus sign) and - (minus sign) followed by a pattern: By default, logs all the lines read from its standard input without modifications; these actions allow selection and deselection of lines, respectively, based on the specified pattern. Because all lines are initially selected, + actions are only effective after a - action that deselects some lines first. The pattern uses the shell's notation for matching strings, and must match whole lines (i.e. partial matching doesn't count), with the following restrictions:
 * Berstein daemontools' only treats asterisks ('*') as special characters in the pattern: they match any string, including the null string, that does not include the next character in pattern instead of the POSIX behaviour of matching the greatest possible number of characters that still allows the remainder of the pattern to match the line, and cannot be escaped with a backslash ('\').
 * Daemontools-encore's can switch between Bernstein daemontools behaviour and using the full range of patterns allowed by POSIX , including backslash escaping. The latter is more flexible at the expense of being less efficient. At the start of logging script action processing, Bernstein daemontools match mode is selected.
 * An F action selects  match mode for all subsequent + and - actions until the next mode-changing action in the logging script.
 * An S action selects Bernstein daemontools match mode for all subsequent + and - actions until the next mode-changing action in the logging script.
 * t: prepends each logged line with '@', followed by a timestamp in external TAI64N format (printed as 24 lowercase hexadecimal characters), and a space. It must be specified as the first action, and any subsequent + and - actions match against lines with the prepended timesptamp. Daemontools-encore also supports a T action, which must also be the first one, that prepends an accustamp-style timestamp and a space instead ( was a program shipped with Bernstein daemontools that was eventually dropped).
 * s followed by an integer: sets the maximum size of the file to the specified value (in bytes), for all subsequent actions in the logging script specifying a logdir, until the next s action.
 * n followed by an integer: sets the maximum number of old log files to the specified value, for all subsequent actions in the logging script specifying a logdir, until the next n action.
 * ! (exclamation mark) followed by a string args: allows to perform some processing of the contents of the  log file during a rotation.  will perform the equivalent of calling sh -c 'args' and feed it the contents of  on its standard input, so args should be the name of a program that can be found via PATH search and its arguments, if any. The .s or .u file produced as a result will contain that program's output.

For details about all actions supported, and about the protocol used for communication between and the processor specified in an ! action, please consult the  man page.

When there's no process running on a logdir, the  file has the executable by user flag set.

Note that the second log file also contains the warning messages. This shows that actions that aren't + or - do not affect the currently selected lines, so warning messages selected by +warning: * stay selected after processing the ./log1 action. To only have the error messages in the log file, a -* action should have immediatly preceded the +error: * action.

Service directory containing a logger:

Note that since makes the service directory its working directory, the relative pathname in 's invocation results in the logging directory being placed wherever the scan directory is.

Resulting supervision tree:

Contents of the logging directory:

This logdir contains two old log files and, since is running, the  file has its executable by user flag cleared.

The TAI64N timestamps can be shown in human-readable form with, for further details please consult the respective man page.

The readproctitle program
The program expects to read a sequence of newline-terminated lines of text from its standard input, and saves them to an automatically rotated log it keeps in memory. The log can be seen in the output of the utility. The number of characters displayed is specified by 's arguments.

is invoked as readproctitle arg1 arg2 ... argn dots. The arguments arg1, arg2, ..., argn can be anything, they are shown as-is in the output of, and can used for displaying some kind of heading that introduces the log. The last argument, dots, must be at least five dots ('.'), and its length specifies the number of characters the log kept in memory will have. In the output of, 's last shown argument will be the contents of the log up to that time instead of dots. Characters shift to the left as reads new lines of text, and older characters are discarded to keep the log of constant size. Therefore, the utility will show each time the most recent data. For further details, please consult the man page.

After enough seconds have elapsed:

The best known use of is in the  script included in Bernstein daemontools and daemontools-encore.

The logging chain
A supervision tree where all leaf processes have a logger can be arranged into what the author of s6 calls the logging chain, which he considers to be technically superior to the traditional syslog-based centralized approach.

Since processes in a supervision tree are created using the POSIX  call, each of them will inherit 's standard input, output and error. A logging chain arrangement using Bernstein daemontools and daemontools-encore is as follows:


 * Leaf processes should normally have a logger, so their standard output and error connect to their logger's standard input. Therefore, all their messages are collected and stored in dedicated, per-service logs by their logger. Some programs might need to be invoked with special options to make them send messages to their standard error, and redirection of standard error to standard output (i.e. 2>&1 in a shell script) must be performed in the servicedir's file.
 * Leaf processes with a controlling terminal are an exception: their standard input, output and error connect to the terminal.
 * , the loggers, and leaf processes that exceptionally don't have logger for some reason, inherit their standard input, output and error from, so their messages are sent wherever the ones from are.
 * Leaf processes that still unavoidably report their messages using  have them collected and logged by a (possibly supervised) syslog server.

Specifying a logger for
Daemontools-encore's allows the pathname of a service directory to be passed as a second argument after the scan directory's pathname. If this argument is present, will launch one  process for the specified directory, and connect its standard output and error to the corresponding supervised process' standard input by a pipe. This makes it possible to set up a logger for in the same way a logger can be set up for the leaf processes of the supervision tree.

Supervised process execution state changes
Bernstein's daemontools and daemontools-encore provide a set of tools for modifying a supervised process' execution state. These tools employ a technique called chain loading by some people, and Bernstein chaining by others. A program prog1 designed to use chain loading is invoked as prog1 arg11 ... arg1n prog2 arg21 ... arg2n, where prog2 is the name of another program. When prog1 is invoked, it performs some action based on arguments arg11, ..., arg1n, and then executes prog2 without creating a new process, using one of the POSIX  calls. Arguments arg21, ..., arg2n are not interpreted by prog1 and are passed along to prog2.

If prog2 is also designed to use chain loading, some of those arguments might in turn be the name of a third program prog3 and its corresponding arguments, which will be executed after prog2 completes its task. As a consequence, it is possible to build a chain of programs that will run one after the other as a single process, therefore preserving the process ID (PID), which makes the technique suitable for the script of a service directory. The final one in such a chain of programs would be the real program intended to be supervised.

The tools provided by Bernstein's daemontools and daemontools-encore to change a process' execution state are:


 * : runs another program with environment modified according to files in a specified directory.
 * : runs another program with environment variables indicating a specified account's user ID and group ID
 * : runs a program in a new session (using POSIX ).
 * : runs another program with a file locked (using Linux  on Gentoo).
 * : runs another program under a specified account's user ID and group ID. Requires root privileges.
 * : runs another program with new soft resource limits (using POSIX ). These limits are set via options specified as 's arguments. Daemontools-encore also allows setting those limits via SOFTLIMIT_* environment variables, applied at  invocation; see environment variables.

For further details on these tools please consult their respective man page.

The directory supplied to the program is called an environment directory (or envdir). Each file in the envdir controls a single environment variable. If the file is empty, and a variable with the same name as the file exists in the process' environment, it is removed from it. If the file is nonempty, the contents of its first line become the value of a new environment variable with the same name as the file. If the variable existed before 's invocation, it is first removed from the environment, and then readded with the new value. Enviroment variables not referred to by a file in the envdir remain unchanged.

This script executes program with effective user daemon and the maximum number of open file descriptors set to 5. This is the same as if performed a   call itself with   set to 5, provided that value does not exceed the corresponding hard limit. As in previous examples, the redirection of stderr to stdout allows setting up a dedicated logger for.

Example service directory with an enviroment directory :

This script adds variables UID, GID and SOFTLIMIT_OPENFILES to the environment, the first two set to the user ID and group ID of account daemon via , and the last one via the enviroment directory , which is used by the invocation to set the maximum number of open file descriptors, provided it is the daemontools-encore version of that program. It is assumed that can use environment variables UID and GID to drop privileges.

Controlling supervised processes
Bernstein daemontools and daemontools-encore provide two programs for controlling a supervised process and to query status information about it: and, respectively.

The program accepts a set of service directory pathnames and options that specify what to do. Some of these options are:


 * svc -d (down): If the supervised process is running, send it a  signal followed by a   signal. The  file won't be reexecuted after that. This is the standard way to manually stop a supervised process. If the process is not running (e.g. because it has a  file or after a previous svc -d command) this is a no operation.
 * svc -u (up): If the supervised process is not running, start it by executing its file. If the process is running, this is mostly a no operation, except that it cancels the effect of a previous svc -o command. This is the standard way to manually start a supervised process that has a  file, or that has been previously stopped with an svc -d command.
 * svc -o (once): If the supervised process is not running, start it by executing its file, but don't restart it if stops. This still allows the process to be monitored and to be reliably sent signals, but it won't actually be supervised. If the process is running, it just asks  to not restart it if it stops. The effect of this option can be cancelled by a subsequent svc -u command, and restored by another svc -o command.
 * svc -x (exit): Asks to exit if the supervised process stops. This is only effective if its  parent stopped running without  being killed, otherwise a new  process will be launched on the next periodic scan.

Other options allow reliably sending signals to a supervised process. For further details, please consult the man page. In particular, svc -a can be used to send a  signal to a supervised  process to force it to perform a rotation.

The program accepts a set of service directory pathnames. Its output is intended to be displayed on a user interface or logged, not to be fed as input to a program for parsing. It displays whether the supervised process is running ('up') or not ('down'), its process ID (PID) if it is up, how long it has been in the current state, whether its current up or down status matches the presence or absence of a down file in the servicedir ('normally up' or 'normally down'), and whether it is transitioning to the desired state or already there ('want up' or 'want down'). The daemontools-encore version also displays the service's extended state (see the, and  files, and the daemontools-encore extended service state), and if there is a  executable file, it will print a second line of output with information about the logger process. If the  option is passed as an argument, the information about the logger process will be suppressed from the output, and if the   option is passed as an argument, only information about the logger process will be displayed, or the message 'no log service' if the servicedir has no  executable file. For further details, please consult the man page.

Bernstein daemontools and daemontools-encore also provide the program, that accepts a service directory pathname, and exits with an exit code of 0 if there is a  process successfully running for that directory, and an exit code of 100 if there isn't. It produces no output if no error was encountered, otherwise it displays an error message and exits with an exit code of 111. Finally, daemontools-encore provides a program, that accepts a service directory pathname, and exits with an exit code of 0 if there is a  process successfully running for that directory and the service's extended state is either started or running, and an exit code of 100 if the service is in any other state, or if there is no  process. It produces no output if no error was encountered, otherwise it displays an error message and exits with an exit code of 111. If there is a executable file,  exits with an exit code of 0 only if both the 'main' service and the logger are in started or running state. If the  option is passed as an argument, only the 'main' service will be checked, and if the   option is passed as an argument, only the logger will be checked. svup -L exits with an exit code of 100 if the servicedir has no executable file. For further details about and, please consult their respective man page.

Service directories with a subdirectory and a  executable file:

Resulting supervision tree:

Note that there are separate processes (with PIDs 2354 and 2355) for test-service1 and its logger, but a single  process (with PID 2356) supervising both test-service2 and its logger.

The status for the test-service1 logger can be seen by passing the pathname of the corresponding subdirectory to :

Starting the supervision tree
In a supervision tree arrangement each process supervises its children, therefore the robustness of the supervision tree ultimately relies on the process. So the problem of how to supervise itself arises, as well as what to do with 's messages in a logging chain.

From OpenRC
Gentoo's packaging of Bernstein daemontools and daemontools-encore provides an OpenRC service script for running, also named. If this service script is used, the scan directory will be. Thus, can be started when the machine boots by adding it to an OpenRC runlevel using :

Or it can also be started manually:

The svscanboot script
Bernstein daemontools and daemontools-encore provide a script called, that can be spawned and supervised by sysvinit by adding a 'respawn' line for it in. is a shell script that launches an process and a  process, with the former's standard output connected to the latter's standard input by a pipe, and every other standard descriptor redirected to. The enviroment will be empty, except for the PATH variable, set to a known value in the script. The scan directory will be.

Used in this way, the supervision tree becomes rooted in process 1, which cannot die without crashing the machine. Also, 's messages will go to, so they can be seen using the utility.

Gentoo users wanting to use will need to manually edit, and then call

This will run when entering runlevels 1 to 5. Because calls  using absolute path, a symlink to the correct path for Gentoo must be created:

The supervise-scripts package
Bruce Guenter's supervise-scripts package provides an  script that also allows sysvinit to launch and supervise. The script will make a backup of (with a name of the form ) and then modify it so that another script,  is run when entering runlevels 1 to 5.

The script is a wrapper around. After some initialization it will replace itself with using the shell's  builtin utility, with its standard input, output and error redirected to. Therefore, 's messages will be shown on the console. The enviroment will be empty, except for the PATH variable, set to a known value in the script. The scan directory will be.

The package also includes a script, that will be called when entering sysvinit's single user mode or runlevels 0, 1 and 6, to perform cleanup. It will stop all lingering processes and their children using an svc -dx command.

Unmerge
All scan directories, service directories, the symlink, etc. must be manually deleted if no longer wanted after removing the package. Also, all modifications to must be manually reverted, even when using the supervise-scripts package, because  does not have an 'undo' option. Lines for, and  must be deleted, and a telinit q command must be used afterwards. If no further changes to have been made after invoking, the backup copy created by that script before the changes are applied can be used to revert them by just replacing  with that file.

External Resources

 * Tokiclover's supervision framework (OpenRC friendly)
 * Avery Payne's supervision-scripts project