Simultaneous Background Process Monitoring in Bash using Multitail

I'm working on an installer for the development server environment we use at www.salsalabs.com

There are time-consuming tasks which are part of the installation process which can be done simultaneously: installation of program dependencies (with brew) and download of code repositories (using git).

I backgrounded the first task in my installation script using the usual & and let the script continue on and complete the second task "normally". After the second task completed, a wait command makes sure that the script does not proceed until the first task is also complete.

That's all well and good. It works just dandy... except that the output of these two processes is intermixed and due to each process providing feedback about the process of individual steps ... that output is very very very very very messy.

So I looked for a solution and I found MultiTail. Awesome! And available via brew! Awesome!

Except ... I don't have log files, and I don't want to generate log files. I want to have two different output streams and use multitail to view them simultaneously in one terminal window.

Two different output streams is easy enough, we've got file descriptors for that, right? (check out the exercise here, it makes my brain hurt). No, we need two different files that are going to act like streams. Oh! We want buffers!

(Cue the superhero music)

Here come FIFOs to the rescue! Use the handy-dandy mkfifo command and you can turn any output stream like stdout, stderr into a buffer that looks like a file to the operating system. Yay!!!!

So, our code, something like this:

install_dependencies.sh &
install_packages.sh
wait

install_something_else.sh

becomes:

mkdir /tmp/fifos
mkfifo "/tmp/fifos/Dependencies"
mkfifo "/tmp/fifos/Code Installation"

install_dependencies.sh > "/tmp/fifos/Dependencies" &
install_packages.sh > "/tmp/fifos/Code Installation"
wait

install_something_else.sh

And we add on multitail:

mkdir /tmp/fifos
mkfifo "/tmp/fifos/Dependencies"
mkfifo "/tmp/fifos/Code Installation"

install_dependencies.sh > "/tmp/fifos/Dependencies" &
install_packages.sh > "/tmp/fifos/Code Installation" &
multitail -ts --basename "/tmp/fifos/Dependencies" "/tmp/fifos/Code Installation"
wait

install_something_else.sh

The problem, now? multitail is never going to exit on it's own, we'll have to press "q" to end the process when the other stuff is done ... assuming we can tell for sure. So we never get to the wait command and we never get on to the next steps of our installation. Let's send an email to the developer and ask him for that feature ... but we'll make our own solution in the meantime.

So, here presented for you is my solution for monitoring a group of parallel background processes using multitail and ending the monitoring when all processes are complete.

PROCESS_COMPLETED_COUNT_TMP_FILE=$(mktemp -t "proc-count")
echo 0 > $PROCESS_COMPLETED_COUNT_TMP_FILE

function killtail {
    MUST_COMPLETE=$1
    PROCESS_COMPLETED=$(cat $PROCESS_COMPLETED_COUNT_TMPO_FILE)
    ((PROCESS_COMPLETED++))

    if [ PROCESS_COMPLETED -ge $MUST_COMPLETE ]; then
        # dangerous if there are multiple multitails running at the same time
        TAIL_PID=$(pidof multitail)
        kill $TAIL_PID
        rm -rf $PROCESS_COMPLETED_COUNT_TMP_FILE
    fi
}

mkdir /tmp/fifos
mkfifo "/tmp/fifos/Dependencies"
mkfifo "/tmp/fifos/Code Installation"

# subshell for everything involved with installing dependencies
(
    install_dependencies.sh > "/tmp/fifos/Dependencies"
    killtail 2
)> "/tmp/fifos/Dependencies" &

# subshell for everything involved with installing code
(
    install_packages.sh > "/tmp/fifos/Code Installation" &
    killtail 2
)> "/tmp/fifos/Code Installation" &

multitail -ts --basename "/tmp/fifos/Dependencies" "/tmp/fifos/Code Installation" 2> /dev/null
wait

rm "/tmp/fifos/Dependencies"
rm "/tmp/fifos/Code Installation"

install_something_else.sh

Now the subshells are backgrounded instead of the scripts themselves. Both "steps" are backgrounded, not just the first one. The command to increment the kill counter and check for the appropriate number of processes to have completed is added to the block of code. As well as backgrounding the subshells, we're directing stdout output from the subshells into our FIFOs, instead of directing the output of individual commands.

We pass the --basename option to multitail so that we see only the name of our FIFO in the multitail windows... and we've used nice human readable names for these temporary buffer files, so isn't that special?

When the kill counter hits the magic number, a SIGTERM will be sent to the multitail that we get back from pidof multitail. This makes multitail error, so we're throwing away the error output from multitail.

Finally, we still have a "wait" statement, just in case our multitail monitoring of the processes doesn't work for some reason. We still want to make sure they complete before moving forward with more steps. We remove the FIFOs and move on to the next steps in our installation.

Happy process monitoring!