denoland / denoland/celld

Poison pill isolation and shuffle sharding

Open
#188 0 comments 9 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
4.6k
Forks
171
PR merge metrics
No merged PRs in 30d

Description

Hi to everyone,

Thanks for the wonderful software. We're already running celld in an almost-prod environment and having our insights.

Current problem is that one cell instance can poison one celld node and this propagates with leases, unfortunately, so our entire cluster is having a bad trip b/c one active cell.

We run thousands of cells, some of which do crazy stupid shi out of our control like `for (;;) {}` and we'd like to be able to isolate such cells to particular nodes if possible - automatically - we are a small team and can't provide babysitting and hyper-caring support to other teams. Long term solution would be to fix our shi but short term we need to be able to isolate such workloads to particular nodes so they don't interfere with the rest of the fleet. I'd rather get my cross-node request timed out, but entire node suffering because one untrusted workload like this.

Also we'd like to make some nodes unavailable until we intentionally move cells there

So a cplane would be great. cellpanel, just like cPanel but 20 years in the future. Even an extension point over stdio or smth like that would do the job, the state is remote so we can point our lb cluster-wide extension and control it. Maybe some control apis already exists, no idea about it. I do cli stuff atm.

P.S. We care deeply about storage usage too, any plans to include that in the rebalance algo that landed in 0.4.1?

What do you think, do we send a patch or it's in your roadmap. I ain't no rusty but can send a patch for polish with your high-level guidance.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by reviewing the rebalance algorithm introduced in 0.4.1 and checking whether control APIs or a stdio extension point already exist; done would require an agreed scope for poison-pill isolation, node availability controls, and storage-aware rebalancing.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.