TestMemProtChg cannot mprotect the stack properly on Linux

_From [reid.kle...@gmail.com](https://code.google.com/u/108532745084733145449/) on February 18, 2011 18:24:43_

I spent a few hours looking into the security-common.TestMemProtChg test failure which I see reliably on my system. The test application writes code to a page and repeatedly changes the protections of it, both for a page in global memory and a page on the stack. The global memory test passes for me, however, the stack page fails. DynamoRIO hits a "found" assertion on line core/linux/os.c:6436 when I run it in debug mode.

After investigating, it seems that DR cannot find the page address in the memory map because it simply isn't there. Below I have the relevant sections of the process page tables. The page DR is looking for starts at 0x7fffffffc000 and is a single 4096 byte page.

Before the first call to protect_mem_check on line 59:
7ffff7ffc000-7ffff7ffd000 r-xp 00000000 00:00 0 [vdso]
7ffff7ffd000-7ffff7fff000 rw-p 0001b000 08:01 818430 /lib/ld-2.7.so
7ffffffea000-7ffffffff000 rw-p 00000000 00:00 0 [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

After the second call to protect_mem_check:

7ffff7ffc000-7ffff7ffd000 r-xp 00000000 00:00 0 [vdso]
7ffff7ffd000-7ffff7fff000 rw-p 0001b000 08:01 818430 /lib/ld-2.7.so
7ffffffea000-7fffffffc000 rw-p 00000000 00:00 0
7fffffffd000-7fffffffe000 r--p 00000000 00:00 0
7ffffffff000-7ffffffff000 rw-p 00000000 00:00 0 [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

Notice that the range from 0x7fffffffc000 to 0x7fffffffd000 is missing, while D to E is present, and E to F is gone. Even though it is missing, I can access the memory from gdb and verify that the code_inc instructions are there:
(gdb) x/3i 0x7fffffffc000
0x7fffffffc000: mov %rdi,%rax
0x7fffffffc003: inc %rax
0x7fffffffc006: retq

Furthermore, this application now segfaults immediately when run not under DR due to the NX bit, as noted in the comments.

What is the proper resolution? Should we delete or disable the test case? I'm mostly curious as to what Linux behavior could cause the page to be missing from the page tables reported to userland, but somehow leave the mapping behind to allow the access.

This probably subsumes the "found" assertion from issue #366 .

_Original issue: http://code.google.com/p/dynamorio/issues/detail?id=387_