huggingface / huggingface/candle

Standardize Interfaces and Plumbing

Open
#3,211 1 comment 1 reaction 0 assignees Claimed by @mayocream View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

The Candle ecosystem appears to require developers to define their own plumbing for inter-process communication and process-level concerns such as logging and error handling.

1. Utilizing standardized abstractions via enums or the like to provide IPC over NCCL/RDMA/TCP/IP/unix/domain/etc sockets would still allow developers to implement new modalities and specific functionality needed for their project while allowing them to use interfaces standard to the ecosystem for interoperability. For example, IPC socket naming is an issue in disag-PD scenarios if the sockets prefix their container hostname to the socket name making P/D unable to find each other in a common namespace. Similar problems exist for distribution of sequential PD calls across containers, hosts, and out to KV cache access across disparate components. Something like [NIXL](https://github.com/ai-dynamo/nixl) may provide a good starting point.
3. Higher level abstraction of API access would also help a ton since projects often have significant differences in how they handle request (chat template) formatting for the final JSON request submitted into the model to tightly couple with the plumbing code facing the model and returning data from it to the system and or the user.

Aside from functional/developer benefits, there are security considerations as well:
1. Should help close the infoleak problem i've noticed in a few projects wherein (especially in distributed modalities) stack traces intended for console output by the server (parent/primary process) end up being returned via the SSE API to clients.
2. Channelize access mechanics to create fulcrums of gating and inspection for attempts to inject control tokens, exfiltrate data, or otherwise abuse the service layer provided.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing linked pull request #3373 and the NIXL project referenced in the issue. The issue names no Candle files or tests, so first map the existing IPC, request-formatting, and SSE error paths before proposing scope. Done would require an agreed design and implementation for standardized plumbing, API access, and inspection boundaries.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai-infra-agents, backend-api-design, distributed-systems, networking, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.