The Unix process API is unreliable and unsafe

B should just always make sure to kill process C before it exits. That way no processes will be orphaned, so no processes will be leaked, and we'll be fine.

Well, what are the possible ways B might exit and need to clean up?

B might choose to exit, possibly by throwing an exception or panicking. In those cases, it's possible for B to kill process C immediately before exiting.

Or B might receive a signal. B might be signaled for conventional reasons, such as a user pressing Ctrl-C, in which case B can still clean up, as long as the programmer or runtime take care to catch every kind of signal.

Or B might be signaled for some more unconventional reasons, such as a segmentation fault. It's still possible for B to clean up in this case, but it may be very tricky to do, and the programmer or runtime may need to take great care to make sure that the pid of C is still accessible even while handling a segfault.

Or B might receive SIGKILL. Unfortunately, this case prevents this solution from working. It's not possible for B to clean up when it receives SIGKILL, so C will be unavoidably leaked.

We might want to say, "never send SIGKILL". But that is impossible, both for a conventional reason and an ironic reason. The conventional reason is that B might have a bug, and hang, and SIGKILL might be the only way to kill it. The ironic reason is that the only way for B to clean up and exit in guaranteed finite time is for it to SIGKILL its own children, so that if they have bugs they will not just hang forever. So B would be SIGKILL'd by its own parent, implementing the same strategy.

So, in summary, it's not possible to guarantee that B cleans up and kills C when it exits, because it might be SIGKILL'd. Even in the case where B isn't SIGKILL'd, it's tricky for a complicated program to always make sure to kill off any child processes when it exits.

We can use the Linux-specific feature PR_SET_PDEATHSIG on process C, to ensure that process C will receive SIGKILL (or another signal) whenever process B exits for any reason, including if process B exits uncleanly due to a bug or SIGKILL.

The issue is that this only works one level down the tree. If C forks off its own process D, the death signal will kill off C but not D.

Extending it to work over an entire tree of processes, requires that the entire tree be using PR_SET_PDEATHSIG (and using it correctly). If we can make that guarantee, this technique will work. But in practice, most large systems can't make that guarantee since they are made up of a large number of programs from many different developers. As one specific example, many applications run subcommands through Unix shells, which don't use PR_SET_PDEATHSIG.

Even in smaller systems where we control all involved programs, this technique isn't perfect, since even programs we control can always have bugs and fail to use PR_SET_PDEATHSIG. We'd prefer a guarantee that relies only on the program at the root of the tree, and doesn't require us to reason about and debug all the programs involved.

B could make sure to always write down the pid of every process it starts, so that we can at least make an attempt to kill any orphaned processes, even if that attempt isn't robust. More generally, B could coordinate with A, and somehow tell A about every process B starts. Then A (which we might trust to be correctly implemented) can handle cleaning up the processes that B starts. This will fail if there's a bug in B, or if B is killed just after starting a process but before telling A, but perhaps it's good enough?

This has the same flaw as PR_SET_PDEATHSIG, in that it only allows for avoiding leaks at a single level. Like PR_SET_PDEATHSIG, all programs involved would need to use our mechanism. And that's infeasible in practice in any large system.

C should just always exit when B does. One common way to do this is for C to read stdin or some other file descriptor and exit as soon as the file descriptor returns EOF; then B can make sure that that file descriptor is the read end of a pipe whose write end only exists in B, which therefore will return EOF as soon as the write end closes, which will happen automatically when B exits.

Unfortunately most Unix programs don't follow any kind of protocol like this; ones which read from stdin do, but of course often stdin is used for other things, like reading from regular files or communicating with other processes in a pipeline.

Again, like with PR_SET_PDEATHSIG, every process in the system would need to be careful to only start children which follow this scheme, and that's infeasible in practice in a large system.

If A runs B inside a Linux container technology, such as a Docker container, then no matter how many processes B starts, A will be able to terminate them all by just stopping the container, and we'll be fine.

Ignoring the other merits of containers, if we're trying to solve the problem of "it is too easy for processes to leak", containers have three main flaws.

1. It's not easy to run a container. Python has a "subprocess.run" function in its standard library, for starting a subprocess. Python has no "container.run" function in its standard library, to start a subprocess inside a container, and in the current container landscape that seems unlikely to change.

   Shell scripts make starting processes trivial, but it's almost unthinkable that, say, bash, would integrate functionality for starting containers, so that every process is started in a container. Leaving aside the issues of which container technology to use, it would be quite complex to implement.

2. Containers require root or running inside a user namespace. The root requirement obviously can't be satisfied by most users. Fortunately, it's possible to start a container without being root by using user namespaces. Unfortunately, user namespaces introduce a number of quirks, such as breaking gdb (by breaking ptrace), so they also can't be used by most users.
3. It's pretty heavyweight to require literally every child process to run in a separate container. Robust usage of pid namespaces (the relevant part of Linux containers) requires that we start up an init process for each pid namespace, separate from the other processes running in the container. This init process will do nothing but increase the load on the system, and it will prevent us from directly monitoring the started processes.

So, running every child process in a separate container isn't a viable solution. We still have no way to easily prevent child processes from leaking.

Process groups and controlling terminals are two features which can be used to terminate a group of processes. Such a group of processes is usually called a "job", since Unix shells use these features and use that terminology. When processes start children, they start out in the same job, and they can all be terminated at once. So if process A put process B in a job, process A could avoid process C leaking by terminating the job.

Unfortunately, neither of these job mechanisms is nestable. If a process puts itself or its children into a new process group or gives itself a new controlling terminal, it completely replaces the old process group or controlling terminal. So that process will no longer be terminated when its original job is terminated!

In other words, if process A puts process B in a job, then process B puts process C in a job, then process B neglects to terminate process C, process C will no longer be in the job that process A knows about, so process C will leak!

So, ironically, if a child process tries to use these features to prevent its own child processes from leaking, it can inadvertantly cause them to leak. This is certainly unsuitable.

Windows 8 added support for nested job objects. Child processes (and all their transitive children) can be associated with a job, and they will all be terminated when the owner of the job exits (or deliberately kills them). Child processes can create their own jobs and assign their own children to those jobs, without interfering with or being aware of their parent job.

Unfortunately, we're using Linux, not Windows. :)

If we run our possibly-malicious process inside a container or virtual machine, then no matter how much it forks and exits, we will be able to terminate the process by just stopping the container (or virtual machine).

This will actually work, to a degree. Most of our earlier concerns (it's too hard and too heavyweight) no longer apply, because in this section we're happy to have any means at all to prevent the attack.

However, it still requires root access (or the use of unprivileged user namespaces) to a run a container or a virtual machine. So this solution is not truly general purpose; we can't use this routinely, every time we create a child process, because our application certainly should not run with root access in the normal case, nor can everything run in an unprivileged user namespace.

We can partially get around the need for root access by having a privileged daemon start processes on our behalf inside a container.

systemd, for example, with its systemd-run API, allows us to request that systemd start up a process for us on the fly. systemd runs every process in a separate cgroup (which is the underlying container mechanism that we would use), so it can protect against the malicious process leak problem.

But having someone else start a process on our behalf breaks a lot of traditional Unix features. For example, we can't easily have our child process inherit stdin/stdout/stderr from us, nor will it inherit environment variables or any ulimits we've placed on ourself. The shell, among other applications, is completely dependent on these features.

Also, this privileged daemon centralizes all the processes we start on the system. We can't, say, set up a truly isolated environment for development or integration testing, because we'll still have to go through the central daemon.

So as a general-purpose mechanism, this is not workable, but it can work in certain constrained scenarios.

What if we limit the number of processes that can exist on the system? Then as the process keeps forking, it will eventually exhaust the available process space and stop, and in that frozen moment of tranquility, an already-started process would be able to kill it.

The number of processes that can exist is actually already limited; there's a maximum pid, and we can't have any more processes than that. The issue is that as processes exit, possibly due to being killed by us, their space is usually freed up, and new processes can be created.

So if the malicious process just keeps forking, it can fill up the space left by previous processes exiting, and this doesn't help us. Stricter limits on the number of processes can prevent fork bombs, but not more general attacks.

However, if we could prevent space from being freed up as processes exit, the space that malicious process has to operate in would shrink and shrink, until finally it is no longer able to fork any more, and we can kill the last copy. Preventing the reuse of process space while under possible attack can be done using a technique that I'll discuss at the end of this article. It's a key part of a robust solution to the process leaking problem.

The kernel could identify processes with some kind of truly globally unique identifier. Then users wouldn't have race conditions when they try to kill them. Note that users can't implement this solution in userspace by tagging the processes with a UUID; it must be a kernel-provided a UUID to avoid the TOCTOU issues.

This would work, but it would be difficult to retrofit onto an existing Unix system: Many applications assume that pids are the same size as 16-bit or 32-bit ints.

We would also pay an efficiency cost, just because of handling a larger identifier. It would be unusual for an operating system to provide references to its internal structures with UUIDs, when it can use more efficient smaller identifiers and provide security through other means.

When process A starts process B, and then process B exits, process A is notified. Furthermore, process B leaves a "zombie process" behind after it exits, which consumes the pid until process A explicitly acts to get rid of the zombie process. These two features allow process A to know exactly when it is safe to send signals to B's pid. So if A stays running for as long as B is running, and only A sends signals to B, we can have signals without races.

This works, and is an excellent replacement for pidfiles. But it doesn't work in all situations.

What if process A exits unexpectedly? Then we are back in the situation of not being able to kill process B without a race condition. Furthermore, what if we genuinely want process B to outlive process A? This is the case whenever we are starting a long-lived process (a daemon), for example.

To support this, instead of forking off a process, process A could send a request to a long-lived supervisor daemon to start process B, as the supervisor daemon's own child. The authors of supervisor daemons such as systemd or supervisord often urge software developers not to fork off their own long-lived processes; instead, say the supervisor daemon authors, we should request that the supervisor daemon fork off long-lived processes on our behalf.

Unfortunately, that has the same issues as discussed in the section on preventing malicious process leaks, where we considered having a privileged daemon create containers on our behalf. We can't easily have our child process inherit stdin/stdout/stderr from us, nor will it inherit environment variables or any ulimits we've placed on ourself. The shell, among other applications, is completely dependent on these features. And this daemon centralizes the processes we start on the system, so it's difficult to set up isolated test or development environments.

Furthermore, even if we have a supervisor daemon starting processes on our behalf, this leaves a static parent-child hierarchy which cannot change. The supervisor daemon cannot, for example, start a new version of itself to upgrade, without careful use of exec, as all of its child processes will stop being its children. Nor can process A initially start up process B as process A's child, and then later decide that process B should live past process A's exit.

What we need is a way to send signals without races, without forcing a specific parent-child hierarchy. If we can make the parent-child hierarchy more flexible, it would work well. We will use this technique in combination with others as part of a full solution at the end of this article.

pidfd was added to Linux in 2019, and solves the issues with using pids to identify processes. Read the LWN summary for more information.

It allows us to break the rigid parent-child hierarchy, and replace it with a more flexible supervision hierarchy, based on passing file descriptors around.

While signalfd is certainly a great help in dealing with signals on Linux, it doesn't actually help deal with the problem of libraries receiving SIGCHLD. You could use signalfd to wait for the SIGCHLD signals, but you still then need to forward the signals to each library.

Can't we just have one library's signal handler call the next library's signal handler?

Rather than explain in this article, I refer the reader to here where it's explained that signal handler chaining can't be done robustly. Libraries have high standard for working, even in strange scenarios!

The issue is that multiple libraries want to handle the task of starting and monitoring children. Can't we just agree on a single standard library that abstracts over SIGCHLD, and have everyone use it? We can provide a file descriptor interface, which is increasingly standard on Linux, and is easy for libraries to use and monitor.

Unfortunately, it would be near impossible to get every other library that wants to use subprocesses or wants to listen for SIGCHLD to use this single standard library.

There are already plenty of libraries which provide wrappers around SIGCHLD/fork/exec, and plenty of code that depends on them. We can't just have a flag day and switch everything over to a new library all at once. This becomes even more tricky in high-level languages, because most languages already come with a higher-level API around spawning processes.

Still, the idea of providing a file descriptor interface for starting and monitoring children is a good one. File descriptors can easily be integrated into an event loop. And a file descriptor can be monitored by a library without interfering with the rest of the program, using a library's own private event loop or other mechanisms. We just need a way to provide that interface that does not interfere with other libraries in the same process.

pidfd allows a library to launch a process and get a pidfd back, then use that pidfd to monitor the process.

A file descriptor can be easily integrated into an event loop, as mentioned in the previous section.