Parallel and Bar

I recently discovered two UNIX command line tools that are really useful. For both tools, I had simple solutions that I've been using for a long time to provide similar functionality. The first tool is GNU Parallel. As the name implies, this tool provides a way to run many commands in parallel. Whats more is that it works in a manner that is consistent with xargs. To get an idea of all the cool things you can do with this tool, I highly recommend looking through the examples in the man page.

Before finding GNU Parallel, I used a simple shell script to accomplish similar tasks. Of course, my version is much more limited in what it can do, it reads one command per line from stdin and runs it in the background. The number of parallel jobs is controlled by using the bash jobs builtin to determine how many tasks are already running. The complete script is shown below.

#!/bin/bash

# Print usage
function usage {
    local prg=`basename $0`
    cat <<USAGE

Usage: $prg [options]

    Options:
    -h                   Print this help message
    -v                   Be verbose with log messages
    -n <tasks>           The number of tasks to run at a time.

USAGE
    exit 1
}

# Check arguments
numProcs=4
verbose="false"
while getopts "n:vh" option; do
    case $option in
        (n) numProcs="$OPTARG" ;;
        (v) verbose="true" ;;
        (h) usage ;;
    esac
done
shift $(($OPTIND-1))

# Read commands from stdin
while read line; do

    # Can we run another?
    while [ "$(jobs -r | wc -l)" -ge "$numProcs" ]; do
        sleep 1
    done

    # Run task
    sh -c "$line" &
done

# Wait for jobs to finish
wait

The second command is the command line progress bar. This command writes progress information to stderr while copying data from stdin to stdout. If you ever need to copy a big file and want some basic stats so you can see that things are still working, this is the tool for you. The output and options are certainly much nicer than the crude C program I was using:

#include <stdio.h>
#include <stdlib.h>
#include <inttypes.h>
#include <unistd.h>

#define PAGESIZE 4096
#define STDIN    0
#define STDOUT   1

int
main() {
    char buffer[PAGESIZE];
    ssize_t length;
    int counter;
    uint64_t bytesWritten = 0;

    while ((length = read(STDIN, buffer, PAGESIZE)) > 0) {
        write(STDOUT, buffer, length);
        bytesWritten += length;
        ++counter;
        if (counter % 10 == 0) {
            fprintf(stderr, "%"PRIu64" bytes\r", bytesWritten);
        }
    }
    fprintf(stderr, "%"PRIu64" bytes\n", bytesWritten);

    return EXIT_SUCCESS;
}

Closing Arbitrary File Descriptors in Bash

While writing a post on file descriptors, I ended up on a tangent trying to answer a question that seemed like it should be simple and obvious. How do you close arbitrary file descriptors in a bash shell script? The chapter on I/O Redirection in the Advanced Bash-Scripting Guide shows the typical examples that you see everywhere. All of these examples use a hardcoded number to identify the descriptor, not a variable. Try to use a variable and bash gets confused thinking the value is the name of a command you wish to execute. For example:

#!/bin/bash
num=5
exec $num>&-

When you run it yields:

$ ./test.sh 
./test.sh: line 3: exec: 5: not found

I eventually found the answer on another blog that had a post about daemonizing bash. The technique is to use the built-in eval command. The variable will be substituted in a string and the string will get executed by eval. It is kind of obvious once I saw how someone else had done it, but for whatever reason it did not occur to me. Since I like examples, here is a script with a closefd function for closing file descriptors:

#!/bin/bash

function closefd {
    local fd=$1
    echo "closing $fd"
    eval "
        exec $fd>&-
        exec $fd<&-
    "
}

function listfds {
    local pid=$1
    local label=$2
    ls /proc/$pid/fd | sort -n | awk '{printf("%s ", $1)}' | sed "s/^/$label: /"
    echo ""
}

# Record the PID of this shell, need to make sure we are tracking fds of this
# shell and not some forked child
pid=$$

# Run commands
listfds $pid "before"
closefd 5
listfds $pid "middle"
closefd 13
listfds $pid "after"

You can run this script with a simple wrapper that opens up a bunch of files that will be inherited when a new process is forked. In my case I used the testclose C program from the forking file descriptors post after removing the call to closefrom. Running the script shows output like:

$ ./testclose closefd.sh
opened 20 files
before: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 255 
closing 5
middle: 0 1 2 3 4 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 255 
closing 13
after: 0 1 2 3 4 6 7 8 9 10 11 12 14 15 16 17 18 19 20 21 22 255 
child 20710 exited with status 0

Forking File Descriptors

When create a new process with a call to fork, one thing the child process will retain is a copy of the parent's open file descriptors:

The child process shall have its own copy of the parent's file descriptors. Each of the child's file descriptors shall refer to the same open file description with the corresponding file descriptor of the parent.

This can be useful, for example, it enables IPC by remapping the standard input, output, and error descriptors to pipes that were created before the fork and then using exec to replace the child image with a new command. However, this behavior can also be an annoyance if one is not careful. An example I hit recently was that the application was occasionally hanging when it shouldn't be. Poking at the system it was blocked on a read to a file descriptor that should have been closed. Debugging further the file descriptor for writing to the pipe was indeed closed in the parent process but, a fork and exec had occurred before it was closed in a different part of the code and the child process still had an open handle to write to that pipe. This failure was of course sporadic and difficult to reproduce because it only occurred if the timing was just right.

Once the problem was diagnosed though, it seemed like it should be a trivial problem to fix. We just need to make sure we close out all of the file descriptors we aren't interested in. In the past I did this using a closefrom system call. This call does not seem to be widely supported though. The naive way to do this is to simply close everything in the range of possible values for file descriptors. On POSIX systems the range can be determined via the _SC_OPEN_MAX setting:

int
closefrom_v1(int fd) {
    int max = sysconf(_SC_OPEN_MAX);
    for (int i = fd; i <= max; ++i) {
        close(i);
    }
    return 0;
}

This method is simple and it basically works, but it bothers me that it is closing lots of file descriptors that aren't open. More importantly though, with this implementation we cannot determine if the close of some descriptors failed. So here is an improved version that checks the error codes and returns -1 on failure and the number of closed descriptors on success:

int
closefrom_v2(int fd) {
    int count = 0;
    int max = sysconf(_SC_OPEN_MAX);
    for (int i = fd; i <= max; ++i) {
        if (close(i) == -1) {
            if (errno != EBADF) {
                return -1;
            }
        } else {
            ++count;
        }
    }
    return count;
}

Notice that we are still closing file descriptors that are not open. I'm not aware of a simple and portable scheme for listing the set of open descriptors for a process so I'll live with the annoyance for now. To make sure it works as expected we can create a simple test program that opens some files, forks a child, has the child use closefrom, and then exec's a process that will list out the file descriptors that are open. Here is the example:

#include <unistd.h>
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/wait.h>

int
closefrom_v1(int fd) {
    int max = sysconf(_SC_OPEN_MAX);
    for (int i = fd; i <= max; ++i) {
        close(i);
    }
    return 0;
}

int
closefrom_v2(int fd) {
    int count = 0;
    int max = sysconf(_SC_OPEN_MAX);
    for (int i = fd; i <= max; ++i) {
        if (close(i) == -1) {
            if (errno != EBADF) {
                return -1;
            }
        } else {
            ++count;
        }
    }
    return count;
}

int
main(int argc, char **argv) {
    if (argc != 2) {
        printf("Usage: %s <cmd>\n", argv[0]);
        exit(1);
    }

    // Open a bunch of files so the child will have something to close
    int numFDs = 42;
    for (int i = 0; i < numFDs; ++i) {
        FILE *f = fopen("/dev/null", "w");
        if (f == NULL) {
            perror("open failed");
            exit(1);
        }
        assert(f != NULL);
    }
    printf("opened %d files\n", numFDs);

    // Fork a child and exec
    pid_t pid = fork();
    if (pid == -1) {
        perror("could not fork");
    } else if (pid == 0) {
        printf("closed %d files\n", closefrom_v2(3));
        if (execlp(argv[1], (char *) 0) == -1) {
            perror("could not exec");
        }
    } else {
        int status;
        waitpid(pid, &status, 0);
        printf("child %d exited with status %d\n", pid, status);
    }
    return 0;
}

For the program to exec, I'll use a simple shell script that lists the contents of the procfs directory /proc/PID/fd:

#!/bin/sh
ls /proc/$$/fd

Note that using procfs to get the listing of filed descriptors is not portable and may not work for you. If you need a more general scheme looking at what lsof is probably a good starting point. Back to the main topic, if the script works you should get output like:

$ ./testclose listfds.sh
opened 42 files
closed 42 files
0  1  2  255
child 31588 exited with status 0

Note file descriptor 255 shown in the listing is added by the shell that is executing the script. It was not open until the after the exec so it was not closed.