[Core][RuntimeEnv] Support per-worker CPU and memory limits for container workers

Description

Ray resource requirements such as num_cpus and memory are currently logical scheduling constraints. They control admission and placement, but they do not limit the operating-system resources used by an individual worker.

This is also true for workers started in a container through RuntimeEnv image_uri. For example, a task requesting num_cpus=0.5 can still use all host CPUs, and a worker can allocate more memory than it requested until a broader pod or node limit is reached.

We would like RuntimeEnv container workers to optionally enforce per-worker CPU and memory limits that are consistent with the resources assigned to the worker lease.

Historical context

Container worker resource limits were previously implemented, but the feature became disconnected during the RuntimeEnv container migration:

• https://github.com/ray-project/ray/pull/16671 introduced start_worker_in_container() and stated that resource limits would be added in a follow-up.
• https://github.com/ray-project/ray/issues/16871 proposed passing allocated resource instances to container worker startup.
• https://github.com/ray-project/ray/pull/17179 implemented an opt-in version with CPU, cpuset, memory, and worker-reuse handling.
• https://github.com/ray-project/ray/pull/19067 moved container startup into the RuntimeEnv Agent. During that migration, the allocated-resource argument was no longer passed to the worker shim. The parsing function was moved but was not connected to the new RuntimeEnv path.
• https://github.com/ray-project/ray/pull/22970 later removed that unreachable parser and its unit test as dead code.
• https://github.com/ray-project/ray/pull/47118 removed the remaining unused allocation field, WorkerCacheKey entry, and feature flag. The discussion suggested adding the feature back once container resource limits were
  actually supported by the current RuntimeEnv architecture.

The current image_uri implementation still contains:

# TODO(chenk008): add resource limit

This history suggests that the old code was removed because it was no longer connected to worker startup, rather than because per-worker container limits were no longer desired.

Desired behavior

For a container worker with resource-limit enforcement enabled:

1. CPU limits should be hard quotas, not only relative CPU shares. Fractional requests such as 0.5, 1.0, and 1.5 CPUs must be preserved.

2. Memory limits should preserve the requested byte value when passed to the container runtime, subject to kernel page-size rounding.

3. A normalized effective limit profile must participate in worker compatibility and idle-worker reuse. A worker started with a one-CPU container limit must not later serve a four-CPU lease. Workers with the same effective profile should remain reusable.

4. If cpuset support is added, CPU IDs must refer to CPUs that are actually available to the Ray node according to the host or parent cgroup. Ray logical resource-instance indexes must not be assumed to be OS CPU IDs.

5. Unsupported environments should fail with a clear error or explicitly fall back according to the agreed policy. In particular, cgroup controller availability and container-runtime permissions should be checked.

6. Non-container workers should not be affected by this feature.

Initially, this could be opt-in while the interaction between Ray's logical resource semantics and operating-system hard limits is evaluated.

Possible implementation direction

This is an internal implementation direction rather than a proposed public API.

After LocalLeaseManager allocates resources for a lease, Ray could normalize the relevant allocation into a typed per-worker limit profile. The profile would then be passed through:

LocalLeaseManager
-> WorkerPool / PopWorkerRequest
-> RuntimeEnv Agent
-> image_uri Podman launch arguments

The profile should be a typed structure rather than the opaque allocation JSON used by the old implementation. For example, it may contain an effective CPU quota, memory bytes, and an optional validated OS cpuset.

WorkerPool should store the profile associated with a started worker and compare it when selecting an idle worker.

RuntimeEnv setup caching should continue to cache the base environment by its serialized RuntimeEnv. A per-worker profile should be applied to a copy of the launch context for each worker request, rather than making the first profile part of the cached RuntimeEnv context.

A first implementation could support CPU quota and memory limits for image_uri Podman workers on cgroup v2. Cpuset and additional platform configurations could be handled separately.

Example

import pathlib
import ray

ray.init()

@ray.remote(
num_cpus=0.5,
memory=256 * 1024 * 1024,
runtime_env={"image_uri": ""},
)
def limits():
files = [
"/sys/fs/cgroup/cpu.max",
"/sys/fs/cgroup/memory.max",
"/sys/fs/cgroup/cpu/cpu.cfs_quota_us",
"/sys/fs/cgroup/cpu/cpu.cfs_period_us",
"/sys/fs/cgroup/memory/memory.limit_in_bytes",
]
return {
path: pathlib.Path(path).read_text().strip()
for path in files
if pathlib.Path(path).exists()
}

print(ray.get(limits.remote()))

For cgroup v2 with a 100000-microsecond CPU period, the expected effective
values would be equivalent to:

cpu.max: 50000 100000
memory.max: 268435456

Equivalent tests should cover 1.0 and 1.5 CPUs.

Acceptance criteria

• Unit tests cover normalized profiles for 0, 0.5, 1.0, and 1.5 CPUs.

• Memory values are passed to the backend as integer bytes.

• The same RuntimeEnv can concurrently start workers with different limit profiles without sharing a profile-specific cached launch context.

• Workers with the same profile can be reused.

• Workers with different CPU or memory profiles cannot be reused.

• Placement-group resource aliases are not double-counted when constructing the profile.

• End-to-end tests verify the cgroup settings inside a Podman worker.

• Unsupported cgroup or permission configurations produce a useful error.

• Existing non-container worker behavior remains unchanged.

Use case

In a shared Ray node, scheduler admission alone does not prevent one worker from consuming CPU or memory beyond its declared resource requirements.

This is especially important for container workers used by independent jobs or teams. A task requesting a fraction of one CPU should not be able to monopolize all CPUs on the node, and one worker should not be able to consume memory reserved for other scheduled work.

Per-worker limits would provide predictable resource usage and noisy-neighbor protection for RuntimeEnv container workloads. This feature is not intended to turn the current container worker implementation into a complete security sandbox.