Last Updated: 2024-04-25

I had the following Procfile

web: rails server -p 3000
docservices: (cd ../oxnotes-docservices && APP_ENV=development bundle exec rackup config.ru)

I planted an un-rescued exception in web.1 (rails server) and the foreman output was as follows:

14:50:52 web.1         | exited with code 1
14:50:52 system        | sending SIGTERM to all processes
14:50:52 docservices.1 | terminated by SIGTERM

• SIGTERM, FYI, is the default signal sent by kill
• Therefore, seeing terminated by SIGTERM for docservices, you'd expect that process to be gone.

However, running ps shows there are remnants still running!

13360 ttys000    0:00.00 sh -c (cd ../oxnotes-docservices && APP_ENV=development bundle exec rackup config.ru)
13361 ttys000    0:00.67 /Users/jack/.rbenv/versions/2.6.4/bin/rackup config.ru

pstree shows these two have a parent child relationship

 |-+- 13360 jack sh -c (cd ../oxnotes-docservices && APP_ENV=development bundle exec rackup config.ru)
 | \--- 13361 jack /Users/jack/.rbenv/versions/2.6.4/bin/rackup config.ru

Let's run it again get more info from ps e.g. process group id pgid

# ask ps for pgid specifically
ps o pid,ppid,pgid,sess,comm,arg

# running Foreman again - unfortunately pids different to above

  PID  PPID  PGID   SESS COMM             ARGS
28590 17478 28590      0 foreman: master  foreman: master
28603 28590 28590      0 puma 4.1.1 (tcp: puma 4.1.1 (tcp://localhost:3000) [oxnotes4]
28604 28590 28590      0 redis-server 127 redis-server 127.0.0.1:6379
28605 28590 28590      0 /Library/Java/Ja /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home
28606 28590 28590      0 sh               sh -c (cd ../oxnotes-docservices && APP_ENV=development bundle e
# It seems odd that these lines are repeated. What I believe is happened is that
# the previous one was `sh -c ""` and the subsequent one is `()`
28608 28606 28590      0 sh               sh -c (cd ../oxnotes-docservices && APP_ENV=development bundle e
28609 28608 28590      0 /Users/jack/.rbe /Users/jack/.rbenv/versions/2.6.4/bin/rackup config.ru
28610 28590 28590      0 /Users/jack/.rbe /Users/jack/.rbenv/versions/2.6.4/bin/rake jobs:work

What do we see?

• foreman itself, where all children share the one PGID, 28590. A process group is a collection of related processes which can all be signalled at once.
• there are THREE entries for Sinatra - 28605, 28608 (both shell commands) and 28509 (actual server). These form a chain of ancestors
• A session is a collection of process groups. Normally, a shell will be a session leader, and every pipeline executed by that shell will be a process group. This is to make it easy to kill the children of a shell when it exits.
• How to kill a process group? killpg $PGID or kill -- -$PGIP
• The (cd ../oxnotes-docservices && y) structure is known as "command grouping". It starts a sub-shell. Compare to {cd ../oxnotes-docservices && y} which instead runs things in the current shell
• When I manually run kill on the topmost process: kill 28606, the similar next process becomes orphaned (PPID=1)
• When I log the processes receiving kill signals, foreman tells me: Killing 76870, 76871, 76872, 76873, 76876

So what's going on with the process staying around after being killed? Basically we have too much indirection with sh -c and () and, since child processes do not get passed along signals, the actual server does not get shut down.

Aside

• From the man page for sh: When bash is interactive, i.e. not here (i.e. not with sh -c) in the absence of any traps, it ignores SIGTERM (so that kill 0 does not kill an interactive shell), and SIGINT is caught and handled (so that the wait builtin is interruptible).

Solutions

• Write an external script that foreman calls (.bin/docservices)

cd ../oxnotes-docservices || echo "DocServices directory not found" || exit
# Keep the same process ID, but run what follows.
# This allows signals to be passed forwards.
exec bundle exec rackup config.ru

Resources

• https://stackoverflow.com/questions/6548823/use-and-meaning-of-session-and-process-group-in-unix
• https://unix.stackexchange.com/questions/18166/what-are-session-leaders-in-ps