lablup / lablup/backend.ai

Investigate and fix permanent hang in netstat_ns(sandbox_key) call

Open
#9,323 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Background:

Analysis of BA-4651 identified that netstat_ns(sandbox_key) calls can permanently hang. While per-scope lock separation (BA-4653) prevents cascading failures, the affected scope (container stat) itself continues to fail.

Root Cause Analysis:

The netstat_ns() function spawns a separate process via ProcessPoolExecutor (or thread pool) that performs:

1. os.open(ns_path) — open the network namespace file

2. setns() — switch to the target namespace

3. psutil.net_io_counters(pernic=True) — read /proc/net/dev

4. setns() — switch back to the original namespace

Multiple scenarios can contribute to the hang:

1. Race condition during container teardown: When a container is shutting down and its network namespace is being torn down, the setns() call may block waiting on the kernel's namespace mutex. Simultaneous namespace destruction and entry can cause kernel-level deadlock.

2. Stale sandbox_key: The SandboxKey path (/var/run/docker/netns/...) obtained from container.show() may no longer be valid but not yet removed from the filesystem. In this case, os.open() succeeds but reading /proc/net/dev after setns() hangs.

3. ProcessPoolExecutor worker hang: When the above operations hang inside a worker process, the run_in_executor() await never completes. ProcessPoolExecutor has no per-task timeout mechanism.

Objective:

- Investigate the root cause of permanent hangs in netstat_ns(sandbox_key) calls

- Implement defensive/recovery logic based on findings (e.g., graceful skip when network namespace is inaccessible, stale sandbox_key detection, etc.)

JIRA Issue: BA-4686

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.