agentscope-ai / agentscope-ai/TuFT
adapter scheduling for DP sampling backend — PR #119 (118 not directly applicable)
- 主要语言
- Python
- 星标
- 69
- 派生
- 14
- 平均合并
- 6 小时 31 分钟
- 30 天内合并 PR
- 13
描述
The recently merged PR #119 (feat: add Data Parallel inference support for small models) implements a data-parallel sampling backend which spawns multiple vLLM replicas and dispatches requests in a round-robin fashion across replicas. This scheduling causes requests that use the same adapter to be dispatched to different instances. Each instance then synchronously loads the same adapter, which is not suitable for multi-tenant scenarios where adapters should be co-located and batched.
Clarification:
- The scheduling implemented in PR #118 is an optimization for the non-DP sampling backend and is NOT directly applicable to the DP sampling backend introduced in PR #119.
- PR #118 currently has some conflicts with PR #119; it cannot be applied to the DP backend without changes.
Current plan:
- Re-architect the request scheduling logic so that it works well in multi-instance Data-Parallel (DP) scenarios and still provides strong adapter coalescing and batching.
- The redesigned scheduler should be adapter-aware in multi-instance DP deployments, minimizing redundant adapter loads across replicas and maximizing batch merge rates.
Suggested approaches to explore:
- Adapter-aware dispatching that routes requests for the same adapter to the same replica when possible.
- Centralized scheduling/coordinator or lightweight consistent-hashing to co-locate adapters and preserve batching.
- Shared adapter cache accessible to replicas, or a fast adapter-sync mechanism that avoids repeated loads.
- Ensure compatibility with DP replicas and avoid introducing cross-replica blocking that would reduce throughput.
Impact:
- Under the current round-robin dispatch, multiple replicas may load the same adapter independently, leading to higher memory usage and redundant work in multi-tenant environments.
- A DP-aware scheduling redesign will be required to avoid this in production multi-tenant deployments.
References:
- PR that introduced DP sampling backend: https://github.com/agentscope-ai/TuFT/pull/119
- PR that implements scheduling optimizations for non-DP backend: https://github.com/agentscope-ai/TuFT/pull/118
(Reported-by: @linqinluli)
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。