kvcache-ai / kvcache-ai/Mooncake

[Bug]: standalone `mooncake_client` does not gracefully drain mounted segments on `SIGTERM`

Open
#3,551 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report

# Bug Report

## Description

We observed two related shutdown problems with standalone `mooncake_client`:

1. When the shutdown cleanup runs, `Client::~Client()` immediately calls `UnmountSegment()` for mounted segments. There is no configurable serving grace period before the segments are removed.

Under high concurrency, peer clients may already have obtained segment or replica information. Removing the segment immediately can cause in-flight or subsequent `get`/`put` operations using that information to fail.

2. In some processes, `kill -TERM ` does not enter the cleanup path at all. The process exits immediately and becomes a zombie, without logging:

```text
Received signal 15, cleaning up resources
```

The standalone signal path uses `pthread_sigmask()` plus a dedicated `sigwait()` thread. Signal masks are per-thread, and we observed several threads with `SIGTERM` unblocked:

```text
main thread: SigBlk=0000000000004203
other thread: SigBlk=0000000000000000
```

If process-directed `SIGTERM` is delivered to an unblocked thread, the default action can terminate the process before the `sigwait()` thread performs cleanup.

The Python console-script launcher is not the cause: it uses `execv()` to replace itself with the native `mooncake_client` ELF, so the resulting process is standalone.

## Steps to reproduce

1. Start standalone `mooncake_client` and mount at least one segment.
2. Send `SIGTERM` to the exact live PID:

```bash
kill -TERM
```

3. Observe the process state and client logs. Other clients may output some error about "Put/Load"

## Actual behavior

Depending on signal delivery:

- cleanup runs but mounted segments are unmounted immediately, without a serving grace period; or
- cleanup is bypassed and the process terminates immediately.

Both behaviors can cause transient failures on peer clients that are still using previously obtained segment information.

## Expected behavior

On `SIGTERM`, standalone `mooncake_client` should reliably enter cleanup, stop accepting new allocations for its segments, keep existing data readable for a configurable grace period, and only then unmount the segments and exit.

## Proposal

- Add a configurable graceful-unmount period for standalone client shutdown.
- During shutdown, request graceful unmount for every mounted segment and keep the process and registered memory alive until the grace period completes or a bounded timeout is reached.
- Make signal handling robust when some process threads do not inherit the blocked signal mask, while keeping cleanup work outside the async signal-handler context.
- Preserve the current immediate-unmount behavior when the grace period is disabled.

### Before submitting...

- [ ] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Research direction

Trace standalone mooncake_client cleanup from Client::~Client() and UnmountSegment(), then inspect the pthread_sigmask() and sigwait() signal path. Verify that SIGTERM reliably reaches cleanup, mounted segments remain available through a configurable grace period or bounded timeout, and disabling the grace period preserves immediate unmounting.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.