Execsnoop can miss some events in parallel environments

It turns out even execsnoop can miss some short-lived processes, when those run in a highly-parallel environment. I've ran into this while building a tree version of execsnoop (for flamegraphs). I have a build system that has quite a few short-lived processes, and sometimes I'd miss a few process, completely breaking the tree chain (which I've rewritten to get the ppid in-kernel).

I've built a simple program to illustrate the issue. It will run a given number of execve, over a given number of processors (it does this by forking).

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>


#define min(a, b) ( (a) < (b) ? (a) : (b))
#define max(a, b) ( (a) > (b) ? (a) : (b))

void execbomb(int children, int start, int end, const char *filename) {
        int span = end - start;
        int per_child = 0, cstart, cend;
        char s_start[32], s_end[32];
        int parent = getpid();
        if (span <= 0)
                return;

        //printf("PID %d Got c=%d s=%d e=%d %s\n", parent, children, start, end, filename);
        if (children > 0) {
                // split the range
                children = min(span, children);
                per_child = max(span/children, 1);
                cstart = start;

                while (children > 0 && cstart <= end) {
                        cend = cstart + per_child;
                        if(!fork()) {
                                // child only works on its slice
                                start = cstart;
                                end = cend;
                                break; //stop forking
                        }
                        children--;
                        cstart += per_child;
                }
                if (parent == getpid()) {
                        if(cend < end)
                                start = cend; // parent gets to finish the job no child had
                        else
                                return;
                }
        }
        sprintf(s_start, "%d", start + 1);
        sprintf(s_end, "%d", end);

        // Fork to generate a new pid every time
        if (!fork())
                execl(filename, filename, s_start, s_end, NULL);
}

int main(int argc, char *argv[]) {
        int children = 0;
        if(argc < 3 || argc > 4) {
                printf("Usage: %s <start> <end> [forks]\n", argv[0]);
                return -1;
        }

        if (argc == 4)
                children = atoi(argv[3]);

        execbomb(children, atoi(argv[1]), atoi(argv[2]), argv[0]);

        return 0;
}

Say you want to run this in a single CPU, and generate 4000 execve():
./execbomb 0 4000 0

In this case, it seems execsnoop.py -n execbomb catches all event. Now, run 17 parallel processes at a given time:
./execbomb 0 4000 17

And then I'm losing about ~274 out of 4000 events. This might need tuning on your hardware, and you can verify easily that the program works properly, like this:

$ strace -e trace=execve -ff ./execbomb 0 4000 17 |& grep 'execve('  |wc -l
4001

(first execve() is the program itself).

Running it through strace does slow it enough so that execsnoop catches all events.

Important point: in these examples, I'm never seeing the perf message "Possibly lost X samples". You could use this program to trigger it though. Also, I always wait for execsnoop.py to finish its processing (through casual top watching). I have disabled the get_ppid() userland search, as well as tried using a bigger perf buffer (page_cnt=512).

If you analyse the log, you might also see events where the arg submission logic fails, with either an empty command line, or multiple command lines concatenated.

Reproduce the loss with the provided execbomb program and the commands in the issue, then inspect execsnoop.py around perf-buffer handling, get_ppid(), and argument submission. Compare parallel runs with the larger page_cnt=512 setting; done means execsnoop captures the events and preserves each command line without concatenation or empties.