lablup / lablup/bssh

feat: Implement real-time streaming output with interactive multi-node UI

Open
#68 0 comments 0 reactions 1 assignee Claimed by @inureyes View on GitHub
priority:medium status:backlog type:enhancement
Dominant language
Rust
Stars
65
Forks
7
Avg merge
1h 30m
Merged PRs (30d)
25

Description

## πŸ“‹ Overview
Implement real-time command output streaming capability for SSH execution, inspired by PR #37 but with enhanced multi-node UI support. ~~Currently, bssh waits for commands to complete before showing any output, making it difficult to monitor long-running operations across multiple nodes.~~

> **Note**: This issue has been significantly updated to reflect the current implementation status as of v1.4.0. Most planned features have been implemented.

## βœ… Implementation Status Summary

| Feature | Status | Location |
|---------|--------|----------|
| Core Streaming API | βœ… Complete | `src/ssh/tokio_client/channel_manager.rs` |
| Independent Stream Management | βœ… Complete | `src/executor/stream_manager.rs` |
| Simple Output Modes (--stream, --output-dir) | βœ… Complete | `src/cli.rs`, `src/executor/output_mode.rs` |
| Interactive TUI (ratatui) | βœ… Complete | `src/ui/tui/` |
| Summary View | βœ… Complete | `src/ui/tui/views/summary.rs` |
| Detail View | βœ… Complete | `src/ui/tui/views/detail.rs` |
| Split View | βœ… Complete | `src/ui/tui/views/split.rs` |
| Diff View | βœ… Complete | `src/ui/tui/views/diff.rs` |
| Progress Parsing | βœ… Complete | `src/ui/tui/progress.rs` |
| TTY Detection | βœ… Complete | `atty` dependency |
| Memory Overflow Protection | βœ… Complete | `RollingBuffer` with 10MB limit |
| Comprehensive Tests | 🚧 Partial | Unit tests exist, more integration tests needed |
| Documentation | 🚧 Partial | Code documented, ARCHITECTURE.md needs update |

## 🎯 Goals (Updated)

1. ~~**Real-time Output Streaming**: Enable streaming of stdout/stderr as commands execute~~ βœ… **Complete**
2. ~~**Multi-node Observability**: Provide dynamic UI to monitor multiple nodes simultaneously~~ βœ… **Complete**
3. ~~**Independent Stream Management**: Maintain separate output streams per node~~ βœ… **Complete**
4. ~~**Backward Compatibility**: Maintain existing API and behavior~~ βœ… **Complete**

## πŸ—οΈ Architecture (Current Implementation)

### Core Streaming Infrastructure βœ…

```rust
// Implemented in src/ssh/tokio_client/channel_manager.rs

/// Command output variants for streaming
#[derive(Debug, Clone)]
pub enum CommandOutput {
StdOut(CryptoVec),
StdErr(CryptoVec),
ExitCode(u32),
}

impl Client {
// Existing method (backward compatible) - uses streaming internally
pub async fn execute(&self, cmd: &str) -> Result;

// Streaming method
pub async fn execute_streaming(
&self,
command: &str,
sender: Sender
) -> Result;

// Sudo password support
pub async fn execute_with_sudo(
&self,
command: &str,
sender: Sender,
sudo_password: &SudoPassword,
) -> Result;
}
```

### Independent Stream Management βœ…

```rust
// Implemented in src/executor/stream_manager.rs

/// Independent output stream for a single node
pub struct NodeStream {
pub node: Node,
receiver: mpsc::Receiver,
stdout_buffer: RollingBuffer, // 10MB max with overflow protection
stderr_buffer: RollingBuffer,
status: ExecutionStatus,
exit_code: Option,
closed: bool,
}

/// Manager for coordinating multiple node streams
pub struct MultiNodeStreamManager {
streams: Vec,
}

/// Execution status for a node's command
pub enum ExecutionStatus {
Pending,
Running,
Completed,
Failed(String),
}
```

### TUI Architecture βœ…

```rust
// Implemented in src/ui/tui/

pub mod app; // TuiApp, ViewMode
pub mod event; // Keyboard event handling
pub mod progress; // Progress bar parsing
pub mod terminal_guard; // RAII terminal cleanup
pub mod views; // summary, detail, split, diff

pub enum ViewMode {
Summary, // Show all nodes status
Detail(usize), // Focus on single node
Split(Vec), // Show multiple nodes in panes
Diff(usize, usize), // Compare two nodes side-by-side
}
```

## πŸ“ Multi-node UI (Implemented)

### CLI Output Modes βœ…

```bash
# TUI Mode (default when TTY detected)
$ bssh -C production "apt-get update"
# β†’ Opens interactive TUI with real-time monitoring

# Stream Mode (--stream flag)
$ bssh -C prod --stream "command"
[node1] Starting process...
[node2] Starting process...
[node1] Progress: 50%

# File Mode (--output-dir flag)
$ bssh -C prod --output-dir ./logs "command"
# Creates: ./logs/node1_TIMESTAMP.stdout, etc.

# Normal Mode (auto when piped)
$ bssh -C prod "uptime" | grep -v idle
```

### TUI Views (All Implemented) βœ…

#### Summary View (Press Esc from other views)
```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Cluster: production - apt-get upgrade β”‚
β”‚ Total: 8 β€’ βœ“ 3 β€’ βœ— 1 β€’ 4 in progress β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ [1] node1 βœ“ Completed (exit: 0) β”‚
β”‚ [2] node2 ⟳ [========= ] 75% β”‚
β”‚ [3] node3 ⟳ Running... β”‚
β”‚ [4] node4 βœ— Exit code: 1 β”‚
β”‚ [5] node5 ⟳ [== ] 25% β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ [1-9] Detail [s] Split [d] Diff [q] Quit [?] Help β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

#### Detail View (Press 1-9)
Full output view with scrolling and follow mode toggle (f key).

#### Split View (Press s)
Shows 2-4 nodes simultaneously in split panes.

#### Diff View (Press d)
Side-by-side comparison of two nodes.

### Keyboard Controls βœ…
- `1-9`: Jump to node detail view
- `s`: Enter split view
- `d`: Enter diff view (select two nodes)
- `f`: Toggle auto-scroll (follow mode)
- `↑/↓`: Scroll output
- `←/β†’`: Switch between nodes in detail view
- `Esc`: Return to summary view
- `?`: Show help overlay
- `q`: Quit

## πŸ“¦ Dependencies (Current)

```toml
# Already implemented in Cargo.toml
ratatui = "0.29" # TUI framework
crossterm = "0.29" # Terminal control
atty = "0.2.14" # TTY detection
tokio = { version = "1.47.1", features = ["full"] }
indicatif = "0.18" # Progress indicators (parallel mode)
```

## πŸ› οΈ Implementation Status

### Task 1: Core Streaming API βœ…
- [x] Implement `execute_streaming()` API
- [x] Add `CommandOutput` enum with StdOut/StdErr/ExitCode
- [x] Implement `CommandOutputBuffer` for internal use
- [x] Ensure `execute()` maintains backward compatibility (uses streaming internally)
- [x] Add error type for `JoinError`
- [x] Add `execute_with_sudo()` for sudo password handling

### Task 2: Independent Stream Management βœ…
- [x] Implement `NodeStream` struct with independent buffering
- [x] Create `MultiNodeStreamManager` for coordinating streams
- [x] Implement non-blocking polling (`poll()` method)
- [x] Add per-node state management (ExecutionStatus)
- [x] Handle partial failures gracefully
- [x] Add `RollingBuffer` with 10MB limit for memory protection

### Task 3: Simple Output Modes βœ…
- [x] Add `--stream` flag for interleaved output with `[node]` prefixes
- [x] Implement `--output-dir` for per-node file output
- [x] Add TTY detection (auto-enable TUI in terminals)
- [x] Update CLI argument parsing in `src/cli.rs`
- [x] Implement `OutputMode` enum (Normal/Stream/File/Tui)

### Task 4: Interactive TUI βœ…
- [x] Add `ratatui` and `crossterm` dependencies
- [x] Implement summary view component
- [x] Add detail view with node switching
- [x] Implement split view mode (2-4 nodes)
- [x] Add diff mode for comparing two nodes
- [x] Implement progress parsing heuristics
- [x] Add keyboard navigation
- [x] Add auto-scroll control (follow mode)
- [x] Implement help overlay (? key)
- [x] Add terminal size validation with error message

### Task 5: Testing & Documentation 🚧
- [x] Unit tests for stream management
- [ ] TUI integration tests with ratatui's test backend
- [ ] Update ARCHITECTURE.md with TUI architecture
- [ ] Add usage examples in README.md

## 🎁 Additional Features Implemented (Beyond Original Scope)

The following features were implemented but not originally planned in this issue:

1. **Sudo Password Support** (`execute_with_sudo()`)
- Automatic sudo prompt detection
- Secure password injection with PTY
- Multiple sudo prompt handling (up to 10 per session)
- Buffer size limits for security (64KB)

2. **Memory Protection**
- `RollingBuffer` with configurable max size (10MB default)
- Automatic old data discard to prevent OOM
- Overflow logging and warnings

3. **Terminal Guard**
- RAII-based terminal cleanup
- Proper handling of crashes/panics
- Cursor visibility management

4. **Progress Parsing Heuristics**
- Detection of `XX%` patterns
- Status message extraction from output
- Integration with summary view

## 🎯 Remaining Work

1. **Documentation**
- [ ] Update ARCHITECTURE.md with new TUI module structure
- [ ] Add TUI screenshots to README
- [ ] Document keyboard shortcuts in help output

2. **Testing**
- [ ] Add TUI snapshot tests using ratatui's test backend
- [ ] Integration tests for streaming execution
- [ ] Performance tests for large output handling

3. **Future Enhancements** (Lower Priority)
- [ ] Configurable buffer sizes via CLI/config
- [ ] Output search/filtering within TUI
- [ ] Session recording and playback
- [ ] Per-node selective logging

## πŸ“ Current Project Structure (Relevant Files)

```
bssh/
β”œβ”€β”€ src/
β”‚ β”œβ”€β”€ cli.rs # CLI with --stream, --output-dir flags
β”‚ β”œβ”€β”€ executor/
β”‚ β”‚ β”œβ”€β”€ mod.rs # ParallelExecutor exports
β”‚ β”‚ β”œβ”€β”€ output_mode.rs # OutputMode enum
β”‚ β”‚ └── stream_manager.rs # NodeStream, MultiNodeStreamManager
β”‚ β”œβ”€β”€ ssh/
β”‚ β”‚ β”œβ”€β”€ client/
β”‚ β”‚ β”‚ β”œβ”€β”€ mod.rs # SshClient exports
β”‚ β”‚ β”‚ β”œβ”€β”€ command.rs # execute(), execute_streaming()
β”‚ β”‚ β”‚ └── ...
β”‚ β”‚ └── tokio_client/
β”‚ β”‚ β”œβ”€β”€ mod.rs # Client, CommandOutput exports
β”‚ β”‚ β”œβ”€β”€ channel_manager.rs # CommandOutput, execute_streaming()
β”‚ β”‚ └── ...
β”‚ β”œβ”€β”€ ui/
β”‚ β”‚ β”œβ”€β”€ mod.rs # UI exports
β”‚ β”‚ └── tui/
β”‚ β”‚ β”œβ”€β”€ mod.rs # run_tui(), TuiExitReason
β”‚ β”‚ β”œβ”€β”€ app.rs # TuiApp, ViewMode
β”‚ β”‚ β”œβ”€β”€ event.rs # Keyboard handling
β”‚ β”‚ β”œβ”€β”€ progress.rs # Progress parsing
β”‚ β”‚ β”œβ”€β”€ terminal_guard.rs # RAII terminal cleanup
β”‚ β”‚ └── views/
β”‚ β”‚ β”œβ”€β”€ mod.rs
β”‚ β”‚ β”œβ”€β”€ summary.rs
β”‚ β”‚ β”œβ”€β”€ detail.rs
β”‚ β”‚ β”œβ”€β”€ split.rs
β”‚ β”‚ └── diff.rs
β”‚ └── commands/
β”‚ └── exec.rs # Execute command with output modes
└── Cargo.toml # ratatui, crossterm, atty dependencies
```

## πŸ“š References

- Original PR #37: https://github.com/lablup/bssh/pull/37
- russh documentation: https://docs.rs/russh/
- ratatui documentation: https://docs.rs/ratatui/
- ratatui examples: https://github.com/ratatui-org/ratatui/tree/main/examples

## βœ… Acceptance Criteria (Status)

- [x] `execute_streaming()` API works with single node
- [x] Multi-node execution maintains independent streams per node
- [x] Node switching is instant (no re-fetching of output)
- [x] Each node preserves scroll position and state when switching
- [x] `--stream` mode works in terminals and pipes
- [x] TUI activates automatically in interactive terminals
- [x] All existing tests pass (backward compatibility)
- [ ] New tests cover streaming scenarios and view modes (partial)
- [ ] Documentation updated (README, ARCHITECTURE.md) (partial)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.