Add signal trap debugging to kernel runner for signal source identification
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Overview
Implement signal trap handling in kernel runner to identify and log the source of signals before termination, providing better debugging capabilities for unexpected kernel shutdowns.
## Background
When kernel runners receive signals and terminate unexpectedly, it's difficult to determine what process or component sent the signal. Adding signal trapping with source identification will help diagnose unexpected terminations and improve system reliability.
## Requirements
### Signal Trap Implementation
1. **Signal Handler Registration**
- Register handlers for common termination signals (SIGTERM, SIGINT, SIGKILL, SIGHUP, etc.)
- Capture signal metadata before shutdown
- Ensure handlers work across different kernel types
1. **Signal Source Identification**
- Log the PID of the process that sent the signal
- Identify the process name if possible
- Record timestamp and signal type
- Capture any additional context available
1. **Logging and Debugging**
- Output detailed information about who sent the signal
- Include stack trace at the time of signal reception
- Ensure logs are flushed before termination
- Store debugging info in a retrievable format
## Technical Design
### Implementation Approach
```python
import signal
import os
import sys
def signal_handler(signum, frame):
# Get sender information if available
sender_pid = os.getppid() # or use other methods to identify sender
# Log signal details
logger.error(f"Received signal {signum} from PID {sender_pid}")
logger.error(f"Signal name: {signal.Signals(signum).name}")
logger.error(f"Stack trace: {traceback.format_stack(frame)}")
# Ensure logs are flushed
sys.stderr.flush()
# Perform cleanup if needed
# Then exit or re-raise signal
```
### Signal Coverage
- SIGTERM (15): Normal termination request
- SIGINT (2): Interrupt from keyboard
- SIGHUP (1): Hangup detected
- SIGQUIT (3): Quit from keyboard
- SIGUSR1/SIGUSR2: User-defined signals
## Implementation Details
- Location: Kernel runner initialization code
- Must work for all kernel types (Python, R, Julia, etc.)
- Consider platform differences (Linux vs macOS signal handling)
- Ensure minimal performance overhead
- Handle signal handler chaining if other handlers exist
## Acceptance Criteria
- [ ] Implement signal trap handlers for major termination signals
- [ ] Log signal source PID and process name when available
- [ ] Include timestamp and full signal information in logs
- [ ] Ensure logs are properly flushed before termination
- [ ] Test with different signal types and sources
- [ ] Verify minimal performance impact
- [ ] Document signal handling behavior
- [ ] Ensure compatibility with existing kernel runner code
## Benefits
- Better debugging of unexpected kernel terminations
- Identify sources of unwanted signals
- Improve system reliability through better diagnostics
- Reduce time to diagnose production issues
- Enable better monitoring and alerting
## Testing Approach
1. Send various signals to running kernels
1. Verify signal source is correctly identified
1. Check log output contains expected information
1. Test with different kernel types
1. Validate graceful shutdown still works properly
JIRA Issue: BA-2530
Contributor guide
Assessment
This issue has not been assessed yet.