[Story] Memory-resources-like reservations

Background

• To make proper use of memory reservations, they need to be tied to the allocations. Then only we can properly account for reserved vs allocated memory.
• Current cuda::*memory_resource concepts/ APIs do not have any extension points to support this ATM.
• The most viable solution is to make memory-resources-like reservations, where a budget gets allocated when a reservation is created and for each allocation and deallocation, reservation is drawn-down or added-back. The changes should also reflected in the allocator that produced the reservation.
• My notes

prototype

• #174 is a prototype of this idea.

device_adaptor adaptor{any_device_resource{rmm::mr::cuda_memory_resource{}}, 1GB};

auto res = adaptor.reserve(4MB, allow_overbooking::NO);

rmm::device_buffer buf{1MB, stream, res.as_device()};
assert(res.balance() == 4MB - 1MB);

auto table = cudf::groupby(..., stream, res.as_device()); // all allocations are capped against res

// The adaptor is a resource too: tracked against available(), but not capped.
rmm::device_buffer scratch{1ull << 20, stream, adaptor};

Feature additions

• Over-reservation policies
• Split a reservation
• OOM policies
• Grow/ shrink reservations
• Reservation release callback
• Sub-reservations/ partitioning an existing reservation (I dont think rapidsmpf/ cucascade uses this currently)

Start by reading prototype issue #174 and the linked notes, then trace the device_adaptor, reserve, as_device, balance, and available concepts shown here. Compare the checked over-reservation policy with the remaining additions: splitting, OOM policies, growing or shrinking, release callbacks, and sub-reservations. Done means the selected reservation features are implemented and their effects are reflected in allocation accounting.