Using a named pipe as a semaphore in a script

At ${work}, again I needed to run a batch of commands and actions multiple times.

Like in a previous post, I have a loop that triggers a lot of actions, using a lot of functions and using many variables, the executing in serial and taking a lot of time. However, the actions do not interfere, running in parallel would save a lot of time in a multicore machine.

Using parallel is out of scope because the subprocesses will not see the function definitions unless I export them all. Even though all the variables need to be exported: painful and dirty.

On the other hand, I’ve been think on another way to do what I want. Maybe I could use use a named pipe as a locking mechanism. Imagine that I want to allow N processes to execute inside the loop. At the beginning the pipe is empty, so the controller would fill the pipe with as N characters as processes would be allowed to run. Then, then main loop will start spawning processes executing the commands I need to run (remember, the & and the parenthesis are your friends here).

So the controller will read a character from the pipe and then spawn a worker. The worker will do its work and once it finishes it will feed a character into the pipe.

If the pipe is not empty, another worker can execute, consuming a character. If the pipe is empty, no more workers can spawn until one of the running ones finishes.

At the end of the process, the pipe will have N characters on it (the initial N character will filled the pipe with).

WARNING

The pipe, by default, is ** BLOCKING ** . That means that the initial write will block, rendering the script useless. To avoid that, the pipe has to be opened RW. That is the purpose of the exec 3<>${TEMP_FIFO}, which is to use the third file descriptor (STDOUT is 0, IN is 1 and ERR is 2) and open it RW (<>).

if [ ! -e ${TEMP_FIFO} ]; then
        mkfifo ${TEMP_FIFO}
        exec 3<>${TEMP_FIFO}
fi

MAX_PROCESSES=25
TOTAL_JOBS=1500

for i in $(seq 1 ${MAX_PROCESSES}); do
        echo -n "0" > ${TEMP_FIFO}

for i in $(seq 1 ${TOTAL_JOBS}); do
        read -u 3 -N 1 x
        (echo "Process $i sleeping"; sleep $[ $RANDOM % 5) +1 ]; echo -n 0 > ${TEMP_FIFO})&
 done

read -u 3 -n MAX_PROCESSES x