Process and MOR for resource reservations by researchers

Thoughts for process and lightweight memorandum of record (MOR) to set expectations.

The goal of this is to increase overall throughput, while minimizing risk of overcrowding the standard partition(s) and maintaining an equitable fair-share environment. Fulfillment of resource reservation requests should not be construed or communicated as granting ongoing special privileges or as a way to circumvent standard resource quotas. It's a way to get researchers with larger-scale needs through the system sooner, gracefully, without negatively impacting others.

Process

1. PI or agent must make request via ServiceNow.
2. PI or agent must supply the following information:
   • Capacity requested. We will allow up to 30% of total capacity at this time.
   • Desired start time for reservation. Start time should be at least 2 business days after submission date.
   • Requested wall-clock time for reservation. We will allow up to 2 weeks continuously at this time. Requests for additional time must be made separately.
   • Approximate number of expected jobs to be run.
   • Evidence of previous runs to support request. This should generally look something like the following, perhaps with more elaboration.

     On average my jobs take 4 hours, and I have 500 jobs to run, so I need about one week with 12 GPUs.

3. Facilitation team will determine occupancy statistics for the past 90 days. The goal here is to have less than 3-sigma risk of exceeding capacity, so less than about 1% chance. This can be found from the std-dev of daily occupancy for 90 days. Use your discretion in tolerating additional risk. @mdefende has details on how to get the statistics.
4. If risk would be exceeded by the request, we can counter-offer fewer GPUs with extended time, up to the time limit.
5. If an agreement is reached, get researcher agreement to MOR below.
6. If they agree, then schedule the reservation: https://github.com/uabrc/devops-docs/issues/36.

MOR

Resource reservations are intended to facilitate completion of computational research tasks that would take excessive time under standard resource quotas, without negatively impacting other researchers.

With that in mind, I understand and agree to all of the following:

• Resource reservation requests are intended to improve equitable access to Research Computing resources.
• My resource reservation is made for a specific use-case and is limited to a specific range of time and a specific collection of resources as agreed in the original Service Request.
• My resource reservation reduces total capacity for other users of the system while it is active.
• Therefore, Research Computing may modify and/or end the reservation early if total resource capacity is exceeded.
• I certify that only one reservation will be active for my research group and project at any given time.

Review the proposed Process and MOR text in this issue, then follow the scheduling reference in issue #36 to understand where the reservation workflow belongs. Confirm how ServiceNow requests, 90-day occupancy statistics, capacity limits, and researcher agreement should be documented. Done means the process and MOR are finalized and published in the appropriate devops-docs location.