ExecutorReservation during task scheduling

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)

````
/// Represents a task slot that is reserved (i.e. available for scheduling but not visible to the
/// rest of the system).
/// When tasks finish we want to preferentially assign new tasks from the same job, so the reservation
/// can already be assigned to a particular job ID. In that case, the scheduler will try to schedule
/// available tasks for that job to the reserved task slot.
#[derive(Clone, Debug)]
pub struct ExecutorReservation {
pub executor_id: String,
pub job_id: Option,
}
````

Instead of making a reservation and cancel the reservation later and depends on the global lock, can we leverage the
compare-and-swap operation provided by sled/etcd?
Since we need to update the available task slots for each executor in atomic, I think compare-and-swap operators can satisfy the needs perfectly. And if we want to switch to an in-memory state store, we can also leverage rust atomic long/int to provide atomicity. Then the assignment logic can be simple.

And I think this statement "When tasks finish we want to preferentially assign new tasks from the same job" is not true.
In a system which has multiple SQL jobs running concurrently(multiple SQL jobs compete for the resources), we should support FAIR or FIFO scheduling , If we prefer to assign new tasks for the same job, it could starve other jobs.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

No repository file or test is named. Start by locating ExecutorReservation and the scheduler's task-slot assignment path, then resolve whether atomic reservations or fair/FIFO scheduling is the intended scope; done requires an agreed design and defined scheduling behavior.

Written by the indexing model from the issue text.