[Question] Wrapping/observability hooks for key operations? / 关于关键操作的包裹式可观测性 hook
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
Your Question
Hi!
We're a downstream consumer of slime and we're exploring adding observability (e.g., tracing) around key operations like training steps, rollout generation, and weight sync. The goal is to understand timing and duration of each phase in the training loop.
The existing --custom-*-path hooks are very useful for replacing or extending behavior, but they fire at a single point — either before or after an operation. For observability, we'd need wrapping semantics (capture both start and end of an operation). As far as we can tell, the only way to achieve this from outside slime would be monkey-patching internal functions like RayTrainGroup.async_train, RolloutManager.generate, update_weights, etc.
We'd prefer to avoid monkey-patching because:
- It depends on internal function names, module paths, and signatures that aren't part of any public contract — these can change without notice across slime versions
- from X import Y style imports in slime create local bindings that bypass module-level patches, requiring us to also patch the importing module's namespace
- Wrapping Ray actor handle methods (e.g., .generate.remote()) requires additional workarounds like blocking with ray.get() and re-putting results, which adds overhead and complexity
Is there a recommended way to instrument slime operations for observability that we might be missing? Or would you be open to considering some form of lightweight wrapping hook at key call-sites in the future?
Completely understand if this is outside the current scope — just wanted to ask before going down the monkey-patching path. Thanks!
你好!
我们是 slime 的下游用户,正在探索为关键操作(如训练步骤、rollout 生成、权重同步等)添加可观测性支持(例如链路追踪),以便了解训练循环中各阶段的时序和耗时。
现有的 --custom-*-path hook 在替换或扩展行为方面非常有用,但它们只在单个时间点触发——操作之前或之后。对于可观测性,我们需要包裹语义(同时捕获操作的开始和结束)。据我们了解,从 slime 外部实现这一点的唯一方式似乎是 monkey-patch 内部函数,如 RayTrainGroup.async_train、RolloutManager.generate、update_weights 等。
我们希望尽量避免 monkey-patching,原因如下:
- 它依赖于不属于任何公开契约的内部函数名、模块路径和签名——这些可能在 slime 版本更新时不经通知而改变
- slime 中 from X import Y 风格的导入会创建本地绑定,绕过模块级别的 patch,导致我们还需要额外 patch 导入方模块的命名空间
- 包裹 Ray actor handle 的方法(例如 .generate.remote())需要额外的变通方案,如通过 ray.get() 阻塞并重新 ray.put() 结果,增加了开销和复杂性
请问是否有我们可能遗漏的推荐方式来对 slime 的操作进行可观测性埋点?或者未来是否会考虑在关键调用点添加某种轻量的包裹式 hook?
完全理解如果这不在当前的协作范围内——只是想在走 monkey-patching 这条路之前先咨询一下。谢谢!
What I've Tried
We've prototyped using functools.wraps to monkey-patch span wrappers onto slime functions at import time, but this approach is brittle across version upgrades and requires additional workarounds for from X import Y bindings and Ray actor handle methods (detailed above).
我们已经尝试使用 functools.wraps 在导入时对 slime 函数进行 monkey-patch 以包裹 span,但这种方式在版本升级时很脆弱,并且需要对 from X import Y 绑定和 Ray actor handle 方法做额外处理(详见上文)。
Environment (if relevant)
No response
Additional Context
No response
Pre-submission Checklist
- I have read the CONTRIBUTING.md and understand the collaboration scope.
- I have read the documentation and FAQ and my question is not answered there.
- I have searched for existing issues and my question has not been asked before.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing --custom-*-path hooks and the named operations RayTrainGroup.async_train, RolloutManager.generate, and update_weights. Determine whether the current extension points support start/end instrumentation; done would be a documented recommendation or a clearly scoped wrapping-hook design.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100