Looking to implement AIO (the old one)

Hello,

Thanks for the great project.

I am interested in debugging a program which requires AIO (the io_submit & co) as a baseline, so rr's current implementation of ENOSYS for it prevents the whole program from starting. (Sidenote - we are in the process of migrating to io_uring, so I may be interested in contributing that too, but that would be later.)

I have a branch of rr where AIO is implemented and works (at least for my program), but I would like to know if my implementation is done in the way that makes most sense for rr.

Also, my implementation currently only supports going through the syscalls, but technically, like io_uring, AIO can also be used in a way where syscalls are not required for gathering completions, by polling at the ring directly from usermode. My guess is that if mainline rr were to support AIO (and no longer return ENOSYS), it would need to support that scenario too, in order to not break things that may be working today by having AIO disabled.

Just for a quick refresher on AIO:

1. You create an AIO context with io_setup. The kernel returns a aio_context_t, which is in fact a usermode pointer to a aio_ring structure. This is a slight difference with io_uring, where you get back a file that you can mmap yourself from usermode.
2. You submit IO with io_submit. Unlike io_uring, there is no way around this, you must always syscall to submit.
3. Get the events:
   3a. Typically, you would io_getevents to park and wait on your queue for completions.
   3b. You may also have the kernel post an event (which you epoll/etc), and upon completion, you either use io_getevents, or peek into the ring yourself.
   3c. You may also never call into the kernel at all and just poll into the ring yourself forever.

So, quite similar to io_uring in multiple ways, but also with notable differences.

So - what I have currently looks like this:

1. On io_setup successful exit, allocate some context that will be responsible for tracking in-flight IO. Let's call this the RemoteAioContext. This object contains a list of iocbs.
2. On io_submit entry, copy the user's iocb structure into the corresponding RemoteAioContext.
3. On io_getevents successful exit, inspect the io_event list, and find the corresponding nodes from the RemoteAioContext.
   3a. If the original opcode was a IOCB_CMD_PREAD or IOCB_CMD_PREADV, call RecordTask::record_remote for the memory, etc.
4. (Also implement io_destroy/io_cancel to cleanup state, etc).

This approach works for my use case, because my program always uses io_getevents, but that may not be the case for other users of AIO. I have read your comments about io_uring implementation from #2613, and I would like to confirm what you think would be the best way to apply this to AIO.

Based on your suggestions for io_uring, what I think could work when applied to AIO, would be to have io_setup allocate its own copy of the aio_ring, and return that back to the debuggee, instead of the real one (this implies that all subsequent syscalls need to tweak the aio_context_t to restore the real value the kernel expects). Then, what I am unsure about, is how rr would go about to migrate completions from the real ring into the copy (while doing the correct record_remote & co if necessary according to opcode). If we assume the "worst" case where the usermode never ever syscalls for completions, when would that migration across rings happen?

Otherwise, would it be a possibility to keep things simple and to require an opt-in option for AIO support, and not implement the polling part?

Review the io_uring implementation discussed in issue #2613 and the AIO syscall entry points named here: io_setup, io_submit, io_getevents, io_destroy, and io_cancel. Confirm the design for syscall completions, direct user-mode ring polling, record_remote handling, and cleanup; done should cover both completion access patterns.