Epic: Intelligent Rate Limiting & Usage Monitoring with ruv-swarm Qudag + SQLite

# Epic: Intelligent Rate Limiting & Usage Monitoring with ruv-swarm Qudag + SQLite

## 🎯 Epic Overview

Implement an intelligent rate limiting and usage monitoring system that leverages ruv-swarm's neural capabilities (Qudag) combined with SQLite for efficient local analytics. This system provides **real-time usage insights**, **optional predictive rate limiting**, and **comprehensive telemetry** using only existing Claude Code capabilities and claude-flow CLI/MCP tools with a Terminal UI interface.

## 📊 Background & Motivation

Based on [Claude Code monitoring documentation](https://docs.anthropic.com/en/docs/claude-code/monitoring-usage), we need:

- **Optional intelligent rate limiting** that adapts to usage patterns
- **Comprehensive usage tracking** for DAU/WAU/MAU analysis
- **Local-first analytics** using SQLite for privacy and performance
- **Neural pattern recognition** to detect anomalies and optimize limits
- **Real-time TUI monitoring** with minimal overhead
- **Claude Code OpenTelemetry integration** using existing capabilities only

## 🏗️ Architecture Diagram

```mermaid
graph TB
subgraph "Claude Code Environment"
CC[Claude Code CLI]
CCH[Claude Code Hooks]
CCE[Claude Code Env Vars]
CCS[Claude Code Settings.json]
end

subgraph "Claude Flow Integration"
CFS[Claude Flow MCP Server]
CFM[CF Monitoring Middleware]
CFC[Claude Flow CLI]
CFT[CF TUI Monitor]
CFR[CF Rate Limiter - OPTIONAL]
CFA[CF Analytics Engine]
end

subgraph "Data Layer"
SQ[SQLite Analytics DB]
RUV[ruv-swarm Neural Analysis]
end

subgraph "Output Layer"
TUI[Terminal UI Display]
OTEL[OpenTelemetry Export]
ALERTS[CLI Alerts]
BLOCKS[Rate Limit Blocks]
end

%% Claude Code to Claude Flow connections
CC --> CCH
CCH --> CFS
CC --> CCS
CCS --> CCE
CCE --> CFM

%% Claude Flow internal flow
CFS --> CFM
CFC --> CFT
CFM --> CFR
CFT --> CFA

%% Data flow
CFS --> SQ
CFR --> SQ
CFA --> SQ
SQ --> RUV

%% Output flow
CFT --> TUI
SQ --> OTEL
RUV --> ALERTS
CFR --> BLOCKS

%% Optional components (dashed style)
style CFR fill:#ffe6e6,stroke:#ff9999,stroke-dasharray: 5 5
style BLOCKS fill:#ffe6e6,stroke:#ff9999,stroke-dasharray: 5 5
style RUV fill:#fff0e6,stroke:#ffb366,stroke-dasharray: 5 5
style ALERTS fill:#fff0e6,stroke:#ffb366,stroke-dasharray: 5 5

%% Core components (solid style)
style TUI fill:#e6f3ff,stroke:#4d94ff
style SQ fill:#e6ffe6,stroke:#66cc66
style CFS fill:#f0e6ff,stroke:#b366ff
```

## 🔧 Claude Code Integration (Verified Capabilities Only)

### 1. Settings.json Configuration (Using Only Supported Options)

**Enhanced `.claude/settings.json` using only documented Claude Code settings:**

```json
{
"env": {
"CLAUDE_FLOW_TELEMETRY_ENABLED": "1",
"CLAUDE_FLOW_RATE_LIMITING": "optional",
"CLAUDE_FLOW_NEURAL_MONITORING": "1",
"CLAUDE_FLOW_TUI_ENABLED": "1",
"OTEL_SERVICE_NAME": "claude-flow",
"OTEL_SERVICE_VERSION": "2.1.0",
"CLAUDE_FLOW_DB_PATH": "~/.claude/claude-flow-analytics.db",
"CLAUDE_FLOW_EXPORT_FORMAT": "otlp,prometheus,local"
},
"hooks": {
"pre-tool": "claude-flow telemetry pre-tool --tool ${TOOL_NAME} --session ${CLAUDE_SESSION_ID:-cli}",
"post-tool": "claude-flow telemetry post-tool --tool ${TOOL_NAME} --success ${SUCCESS} --duration ${DURATION} --session ${CLAUDE_SESSION_ID:-cli}",
"pre-edit": "claude-flow monitor track-edit --file ${FILE_PATH}",
"post-edit": "claude-flow monitor track-edit-complete --file ${FILE_PATH} --lines-changed ${LINES_CHANGED}"
},
"permissions": {
"allow": ["*"],
"additionalDirectories": ["~/.claude/claude-flow-data"]
}
}
```

### 2. Environment Variable Configuration (Claude Code Supported)

**Using only documented Claude Code environment variables:**

```bash
# Standard Claude Code variables
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export DISABLE_TELEMETRY=0

# Claude Flow specific variables (via env in settings.json)
export CLAUDE_FLOW_TELEMETRY_ENABLED=1
export CLAUDE_FLOW_RATE_LIMITING=optional # Can be: disabled, optional, enforced
export CLAUDE_FLOW_NEURAL_MONITORING=1
export CLAUDE_FLOW_TUI_REFRESH_INTERVAL=5000
export CLAUDE_FLOW_ANALYTICS_RETENTION_DAYS=90
```

### 3. Hooks Integration (Using Existing Claude Code Hooks)

**Pre-tool hook (`claude-flow telemetry pre-tool`):**

```bash
#\!/bin/bash
# Using Claude Code's documented hook system
TOOL_NAME="$1"
SESSION_ID="${CLAUDE_SESSION_ID:-cli-session}"
START_TIME=$(date +%s%3N)

# Store timing for post-hook
echo "$START_TIME" > "/tmp/claude-flow-${TOOL_NAME}-$$"

# Optional rate limiting check (only if enabled)
if [ "$CLAUDE_FLOW_RATE_LIMITING" = "enforced" ] || [ "$CLAUDE_FLOW_RATE_LIMITING" = "optional" ]; then
claude-flow rate-limit check \
--tool "$TOOL_NAME" \
--user "$(whoami)" \
--session "$SESSION_ID" \
--mode "$CLAUDE_FLOW_RATE_LIMITING"
fi

# Record pre-tool telemetry
claude-flow telemetry record \
--event "tool_start" \
--tool "$TOOL_NAME" \
--timestamp "$START_TIME" \
--session "$SESSION_ID"
```

**Post-tool hook (`claude-flow telemetry post-tool`):**

```bash
#\!/bin/bash
TOOL_NAME="$1"
SUCCESS="$2"
SESSION_ID="${CLAUDE_SESSION_ID:-cli-session}"
END_TIME=$(date +%s%3N)
START_TIME=$(cat "/tmp/claude-flow-${TOOL_NAME}-$$" 2>/dev/null || echo "$END_TIME")
DURATION=$((END_TIME - START_TIME))

# Clean up
rm -f "/tmp/claude-flow-${TOOL_NAME}-$$"

# Record completion telemetry
claude-flow telemetry record \
--event "tool_complete" \
--tool "$TOOL_NAME" \
--success "$SUCCESS" \
--duration "$DURATION" \
--session "$SESSION_ID"

# Background neural analysis (optional)
if [ "$CLAUDE_FLOW_NEURAL_MONITORING" = "1" ]; then
claude-flow neural analyze-usage \
--tool "$TOOL_NAME" \
--duration "$DURATION" \
--success "$SUCCESS" \
--background &
fi
```

## 💻 Terminal UI (TUI) Monitor

### Real-time TUI Interface (No Web Dashboard)

```javascript
// src/tui/claude-flow-monitor.js
import blessed from 'blessed';
import { ClaudeFlowAnalytics } from '../analytics/analytics-engine.js';

class ClaudeFlowTUIMonitor {
constructor() {
this.screen = blessed.screen({
smartCSR: true,
title: 'Claude Flow Analytics Monitor'
});

this.analytics = new ClaudeFlowAnalytics();
this.refreshInterval = parseInt(process.env.CLAUDE_FLOW_TUI_REFRESH_INTERVAL) || 5000;
this.setupLayout();
}

setupLayout() {
// Header with key metrics
this.headerBox = blessed.box({
top: 0,
left: 0,
width: '100%',
height: 3,
content: 'Claude Flow Analytics Monitor - Press q to quit',
tags: true,
border: { type: 'line' },
style: { border: { fg: 'cyan' } }
});

// Live metrics panel
this.metricsBox = blessed.box({
top: 3,
left: 0,
width: '50%',
height: '40%',
label: 'Live Metrics',
tags: true,
border: { type: 'line' },
style: { border: { fg: 'green' } }
});

// Rate limiting status (if enabled)
this.rateLimitBox = blessed.box({
top: 3,
left: '50%',
width: '50%',
height: '40%',
label: 'Rate Limiting (Optional)',
tags: true,
border: { type: 'line' },
style: { border: { fg: 'yellow' } }
});

// Recent activity log
this.activityBox = blessed.log({
top: '43%',
left: 0,
width: '100%',
height: '40%',
label: 'Recent Activity',
tags: true,
border: { type: 'line' },
scrollable: true,
mouse: true,
style: { border: { fg: 'blue' } }
});

// Neural insights panel
this.neuralBox = blessed.box({
top: '83%',
left: 0,
width: '100%',
height: '17%',
label: 'Neural Insights',
tags: true,
border: { type: 'line' },
style: { border: { fg: 'magenta' } }
});

// Add all to screen
this.screen.append(this.headerBox);
this.screen.append(this.metricsBox);
this.screen.append(this.rateLimitBox);
this.screen.append(this.activityBox);
this.screen.append(this.neuralBox);

// Key bindings
this.screen.key(['escape', 'q', 'C-c'], () => process.exit(0));
this.screen.key(['r'], () => this.refresh());
}

async refresh() {
await this.updateMetrics();
await this.updateRateLimits();
await this.updateActivity();
await this.updateNeuralInsights();
this.screen.render();
}

async updateMetrics() {
const metrics = await this.analytics.getLiveMetrics();

const content = [
`{bold}Active Sessions:{/bold} ${metrics.activeSessions}`,
`{bold}Operations/min:{/bold} ${metrics.operationsPerMinute}`,
`{bold}Avg Response Time:{/bold} ${metrics.avgResponseTime}ms`,
`{bold}Success Rate:{/bold} ${metrics.successRate}%`,
`{bold}Active Swarms:{/bold} ${metrics.activeSwarms}`,
`{bold}Total Agents:{/bold} ${metrics.totalAgents}`,
`{bold}Tokens Used (24h):{/bold} ${metrics.tokensUsed}`,
`{bold}Est. Cost (24h):{/bold} $${metrics.estimatedCost}`
].join('\n');

this.metricsBox.setContent(content);
}

async updateRateLimits() {
const rateLimitMode = process.env.CLAUDE_FLOW_RATE_LIMITING || 'disabled';

if (rateLimitMode === 'disabled') {
this.rateLimitBox.setContent('{yellow-fg}Rate limiting disabled{/yellow-fg}');
return;
}

const limits = await this.analytics.getRateLimitStatus();

const content = [
`{bold}Mode:{/bold} ${rateLimitMode}`,
`{bold}Blocked Requests (1h):{/bold} ${limits.blockedRequests}`,
`{bold}Active Limits:{/bold} ${limits.activeLimits}`,
`{bold}Neural Adjustments:{/bold} ${limits.neuralAdjustments}`,
'',
'{bold}Current Limits:{/bold}',
...limits.currentLimits.map(l => ` ${l.type}: ${l.current}/${l.max}`)
].join('\n');

this.rateLimitBox.setContent(content);
}

async updateActivity() {
const recentActivity = await this.analytics.getRecentActivity(20);

recentActivity.forEach(activity => {
const timestamp = new Date(activity.timestamp).toLocaleTimeString();
const status = activity.success ? '{green-fg}✓{/green-fg}' : '{red-fg}✗{/red-fg}';

this.activityBox.log(
`${timestamp} ${status} ${activity.operation} (${activity.duration}ms)`
);
});
}

async updateNeuralInsights() {
if (process.env.CLAUDE_FLOW_NEURAL_MONITORING \!== '1') {
this.neuralBox.setContent('{yellow-fg}Neural monitoring disabled{/yellow-fg}');
return;
}

const insights = await this.analytics.getNeuralInsights();

const content = [
`{bold}Pattern Classification:{/bold} ${insights.patternType} (${insights.confidence}%)`,
`{bold}Anomalies Detected (1h):{/bold} ${insights.anomaliesDetected}`,
`{bold}Usage Prediction:{/bold} ${insights.usagePrediction}`,
`{bold}Optimization Suggestions:{/bold} ${insights.suggestions.join(', ')}`
].join('\n');

this.neuralBox.setContent(content);
}

start() {
this.refresh();
setInterval(() => this.refresh(), this.refreshInterval);
this.screen.render();
}
}

export { ClaudeFlowTUIMonitor };
```

## 🛡️ Optional Rate Limiting System

### Configurable Rate Limiting (Disabled by Default)

```javascript
// src/rate-limiting/optional-rate-limiter.js
class OptionalRateLimiter {
constructor() {
this.mode = process.env.CLAUDE_FLOW_RATE_LIMITING || 'disabled';
this.enabled = this.mode \!== 'disabled';
this.analytics = new ClaudeFlowAnalytics();
}

async checkLimit(userId, operation, options = {}) {
// If rate limiting is disabled, always allow
if (this.mode === 'disabled') {
return { allowed: true, reason: 'Rate limiting disabled' };
}

// Get current usage
const usage = await this.analytics.getUserUsage(userId, '1h');

// Get base limits for operation type
const baseLimit = this.getBaseLimit(operation);

if (this.mode === 'optional') {
// Optional mode: warn but don't block
if (usage.count >= baseLimit) {
console.warn(`⚠️ Rate limit recommendation: ${operation} usage is high (${usage.count}/${baseLimit})`);
return {
allowed: true,
warning: true,
reason: `Optional rate limit exceeded: ${usage.count}/${baseLimit}`
};
}
} else if (this.mode === 'enforced') {
// Enforced mode: block when limit exceeded
const adaptedLimit = await this.calculateAdaptiveLimit(userId, operation, baseLimit);

if (usage.count >= adaptedLimit.limit) {
return {
allowed: false,
reason: `Rate limit exceeded: ${usage.count}/${adaptedLimit.limit} (${adaptedLimit.reason})`
};
}
}

return { allowed: true, reason: 'Within limits' };
}

async calculateAdaptiveLimit(userId, operation, baseLimit) {
// Only use neural analysis if enabled
if (process.env.CLAUDE_FLOW_NEURAL_MONITORING \!== '1') {
return { limit: baseLimit, reason: 'Base limit (neural disabled)' };
}

try {
// Get neural pattern analysis
const userHistory = await this.analytics.getUserHistory(userId, '24h');
const pattern = await this.analyzeUsagePattern(userId, userHistory);

// Adaptive adjustment based on pattern
const multiplier = pattern.patternType === 'normal' ? 1.2 :
pattern.patternType === 'burst' ? 0.8 : 0.5;

return {
limit: Math.floor(baseLimit * multiplier),
reason: `Neural adaptive: ${pattern.patternType} (${pattern.confidence}%)`
};
} catch (error) {
console.error('Neural analysis failed, using base limit:', error);
return { limit: baseLimit, reason: 'Base limit (neural failed)' };
}
}

getBaseLimit(operation) {
const baseLimits = {
'mcp_tool': 60, // 60 MCP calls per hour
'swarm_spawn': 10, // 10 swarm spawns per hour
'neural_train': 5, // 5 neural training sessions per day
'cli_command': 300 // 300 CLI commands per hour
};

return baseLimits[operation] || baseLimits['cli_command'];
}
}
```

## 📊 CLI Commands for Monitoring

### New Claude Flow CLI Commands

```bash
# Start TUI monitor
claude-flow monitor tui
claude-flow monitor tui --refresh-interval 3000

# Analytics commands
claude-flow analytics usage --user --timeframe 24h
claude-flow analytics swarms --performance --timeframe 7d
claude-flow analytics costs --breakdown team,project

# Rate limiting commands (optional)
claude-flow rate-limits status
claude-flow rate-limits configure --mode optional < /dev/null | enforced|disabled
claude-flow rate-limits test --user $(whoami) --operation mcp_tool

# Telemetry commands
claude-flow telemetry export --format otlp --endpoint http://localhost:4317
claude-flow telemetry status
claude-flow telemetry configure --exporters prometheus,local

# Neural analysis commands (optional)
claude-flow neural patterns --analyze --user $(whoami)
claude-flow neural insights --anomalies --timeframe 1h
claude-flow neural optimize --dry-run
```

## 💾 SQLite Schema (Verified for Local Storage)

```sql
-- Core usage tracking
CREATE TABLE claude_flow_usage (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
user_id TEXT NOT NULL,
session_id TEXT,
operation_type TEXT, -- 'mcp_tool', 'cli_command', 'swarm_operation'
operation_name TEXT,
duration_ms INTEGER,
success BOOLEAN,
error_message TEXT,
rate_limit_status TEXT, -- 'allowed', 'warned', 'blocked'
neural_score REAL
);

-- Rate limiting events (optional table)
CREATE TABLE claude_flow_rate_limits (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
user_id TEXT NOT NULL,
operation_type TEXT,
limit_type TEXT, -- 'hourly', 'daily'
current_count INTEGER,
limit_value INTEGER,
action_taken TEXT, -- 'allowed', 'warned', 'blocked'
neural_adjustment REAL
);

-- Session analytics
CREATE TABLE claude_flow_sessions (
session_id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
start_time DATETIME,
end_time DATETIME,
total_operations INTEGER DEFAULT 0,
successful_operations INTEGER DEFAULT 0,
total_duration_ms INTEGER DEFAULT 0,
swarm_count INTEGER DEFAULT 0,
agent_count INTEGER DEFAULT 0
);

-- Neural insights (optional table)
CREATE TABLE claude_flow_neural_insights (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
user_id TEXT NOT NULL,
pattern_type TEXT, -- 'normal', 'burst', 'anomaly'
confidence_score REAL,
recommendations JSON,
timeframe TEXT
);
```

## 🚀 Implementation Roadmap

### Phase 1: Foundation (Week 1-2)
- [ ] **Claude Code Hooks Integration**: Implement pre/post tool hooks using documented APIs
- [ ] **Environment Variables**: Set up CLAUDE_FLOW_* variables via settings.json env
- [ ] **SQLite Analytics**: Basic usage tracking with local database
- [ ] **Optional Rate Limiting**: Configurable rate limiting (disabled by default)

### Phase 2: TUI & Analytics (Week 3-4)
- [ ] **Terminal UI Monitor**: Real-time TUI interface using blessed.js
- [ ] **CLI Analytics Commands**: Basic analytics and reporting commands
- [ ] **OpenTelemetry Export**: Standard OTel export using existing Claude Code telemetry
- [ ] **Settings Validation**: Ensure all settings use only documented Claude Code options

### Phase 3: Intelligence Layer (Week 5-6)
- [ ] **Neural Pattern Analysis**: Optional ruv-swarm integration for usage patterns
- [ ] **Adaptive Rate Limiting**: Neural-driven limit adjustments (when enabled)
- [ ] **Anomaly Detection**: Optional intelligent alerting via CLI
- [ ] **Cost Attribution**: Token estimation and cost tracking

### Phase 4: Production Features (Week 7-8)
- [ ] **Advanced TUI Features**: Enhanced terminal interface with filtering/sorting
- [ ] **Export Capabilities**: Multi-format export (OTel, Prometheus, JSON)
- [ ] **Configuration Validation**: Comprehensive settings.json validation
- [ ] **Documentation**: Complete integration guides and examples

## ✅ Claude Code Compatibility Verification

### Verified Compatible Features:
- ✅ **settings.json env variables**: Documented Claude Code feature
- ✅ **hooks (pre-tool, post-tool)**: Documented Claude Code feature
- ✅ **permissions.additionalDirectories**: Documented Claude Code feature
- ✅ **OpenTelemetry integration**: Documented Claude Code telemetry
- ✅ **CLI tool execution**: Standard Claude Code tool pattern

### Optional Features (User Configurable):
- ⚠️ **Rate limiting**: Optional, disabled by default
- ⚠️ **Neural analysis**: Optional, configurable via environment variables
- ⚠️ **Advanced analytics**: Optional, uses only local SQLite storage

### Not Used (Removed):
- ❌ **Custom settings.json sections**: Only using documented env/hooks/permissions
- ❌ **Web dashboard**: Replaced with TUI
- ❌ **Custom Claude Code APIs**: Only using documented capabilities

## 🔗 Related Issues

- Issue #262: Complete ruv-swarm neural integration
- Claude Code telemetry standardization
- Settings.json compatibility verification
- TUI interface development

## 🏷 Labels

`enhancement`, `agents`, `v2.0.0`

---

**Epic Owner:** @ruvnet
**Estimated Effort:** 8 weeks
**Target Release:** v2.1.0
**Priority:** High

This epic provides intelligent monitoring and optional rate limiting using only verified Claude Code capabilities, with a clean TUI interface and complete configurability.

The proposal names src/tui/claude-flow-monitor.js and src/rate-limiting/optional-rate-limiter.js, alongside Claude Code settings and telemetry hooks. Start by checking whether these entry points and the documented hook capabilities exist in the repository. Done would require a defined, tested scope for telemetry, SQLite analytics, the TUI, neural monitoring, and optional rate limiting, but the issue does not identify tests or a smaller deliverable.

Written by the indexing model from the issue text.