rr record seems to loop infinitely in 32-bit x86 with certain system calls

I don't have a full diagnosis of what is going wrong here. The symptoms I've observed are that attempts to run rr-record on an x86 32-bit build of OpenJDK tip do not return from a call to the mmap system call. top reports that rr continue to run, but trace statements placed before and after the mmap call confirm that control does not return from mmap.

When I run the same 32-bit build outside rr, trace statements confirm that the mmap call is successful. I have also been able to step through (over) the call to mmap service with gdb.

This problem with mmap calls is not universal. I have found that during initialization of the JVM, the first 7 calls to mmap succeed. As far as I have been able to understand, all of the invocations are identical except for the size of the request:

With system call prototype: void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset)
The successful calls are parameterized, in order, as:

mmap(nullptr, 4096, PROT_NONE, MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS, -1, 0);
mmap(nullptr, 8192, PROT_NONE, MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS, -1, 0);
mmap(nullptr, 32768, PROT_NONE, MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS, -1, 0);
mmap(nullptr, 251_658_240, PROT_NONE, MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS, -1, 0);
mmap(nullptr, 45_056, PROT_NONE, MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS, -1, 0);
mmap(nullptr, 962_560, PROT_NONE, MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS, -1, 0);
mmap(nullptr, 962_560, PROT_NONE, MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS, -1, 0);

The eight call, which has the largest requested size so far, is the one that fails to return (when running in rr):

mmap(nullptr, 1_073_741_824, PROT_NONE, MAP_PRIVATE | MAP_NORESERVE | MAP_ANONYMOUS, -1, 0);

Maybe this description is enough to prompt some thought about what might be going wrong. If not, I can provide more details regarding how to reproduce this problem and/or I can gather more details within my environment to help you debug futher.

Thanks.

Start by reproducing rr-record with the 32-bit OpenJDK build and compare the eighth mmap call with the successful calls, using the reported gdb stepping and trace statements. Done means identifying why the 1 GiB mmap does not return under rr and documenting or implementing a fix that allows recording to continue.