Starship 2.0

# Starship 2.0: Migrate from Helm to KubernetesJS Architecture

## Epic Overview

This epic tracks the migration of Starship from a Helm-based deployment system to a modern KubernetesJS-based architecture that provides better programmatic control, improved debugging capabilities, and a more maintainable codebase.

## Problem Statement

### Current Architecture Limitations

1. **Shell Command Dependency**: The current client heavily relies on shell execution of `kubectl`, `helm`, and `docker` commands, creating:
- **Debugging difficulties**: Errors are opaque and hard to trace
- **Platform compatibility issues**: Different behavior across operating systems
- **Limited error handling**: Shell command failures provide minimal context
- **Testing challenges**: Mocking shell commands is complex and unreliable

2. **Helm Template Complexity**: The current Helm-based approach:
- **Limited programmatic control**: Difficult to dynamically modify resources
- **Complex templating logic**: Helm templates in `starship/charts/devnet/templates/` are hard to maintain
- **No intermediate artifacts**: Resources are deployed directly without visibility into generated YAML
- **Debugging overhead**: Requires `--dry-run --debug` to see generated resources

3. **Deployment Opacity**:
- **No YAML inspection**: Users cannot see what will be deployed before execution
- **Limited customization**: Hard to modify resources programmatically before deployment
- **No staged deployment**: All resources deployed at once without granular control

### Current Implementation Details

Based on codebase analysis:

- **Main client**: `clients/js/packages/client/src/client.ts` (999 lines)
- **Shell dependencies**: Uses `shelljs` for executing external commands
- **Helm charts**: Located in `starship/charts/devnet/` with complex template structure
- **Dependencies**: `kubectl`, `helm`, `docker` must be installed and configured
- **CLI commands**: `deploy`, `start`, `stop`, `undeploy` all rely on shell execution

```typescript
// Current approach - shell-based
this.exec(['helm', 'install', '-f', this.ctx.config, this.config.name, this.ctx.chart]);
this.exec(['kubectl', 'get', 'pods', ...this.getArgs()]);
```

## Proposed Solution: KubernetesJS Architecture

### Vision

Transform Starship into a modern, programmatic Kubernetes deployment system that:

1. **Generates YAML artifacts first** for inspection and version control
2. **Uses KubernetesJS** for direct API communication
3. **Provides granular control** over deployment lifecycle
4. **Eliminates shell dependencies** for core functionality

### Architecture Overview

```mermaid
graph TD
A[Starship Config] --> B[KubernetesJS Templating Engine]
B --> C[Generated YAML Files]
C --> D[YAML Validation & Review]
D --> E[KubernetesJS Client Deployment]
E --> F[Direct Kubernetes API Calls]
F --> G[Deployed Resources]

H[Port Forwarding] --> I[KubernetesJS Port Forward API]
J[Resource Monitoring] --> K[KubernetesJS Watch API]
```

## Implementation Proposals

### Proposal 1: Phased Migration (Recommended)

**Phase 1: YAML Generation Layer**
- Create KubernetesJS templating system parallel to Helm
- Generate YAML files to disk before deployment
- Maintain Helm deployment for backward compatibility
- Add `--generate-only` flag for YAML-only mode

**Phase 2: KubernetesJS Deployment**
- Implement KubernetesJS-based deployment alongside YAML generation
- Add `--use-k8s-client` flag for new deployment method
- Maintain shell-based deployment as fallback

**Phase 3: Shell Command Elimination**
- Replace all `kubectl` commands with KubernetesJS API calls
- Remove Helm dependency entirely
- Update dependency management (`deps.ts`) to remove helm requirement

**Phase 4: API Enhancement**
- Add advanced features like resource watching, streaming logs
- Implement proper error handling and retry logic
- Add deployment rollback capabilities

### Proposal 2: Complete Rewrite

**Advantages:**
- Clean architecture from start
- No technical debt from old system
- Optimized for KubernetesJS patterns

**Disadvantages:**
- Higher risk due to big-bang approach
- Temporary loss of Helm-specific features
- Longer development time before usable release

### Proposal 3: Hybrid Approach

**Concept:**
- Keep Helm for complex templating scenarios
- Use KubernetesJS for deployment and management
- Generate YAML from Helm, deploy via KubernetesJS

**Trade-offs:**
- Maintains Helm dependency
- More complex architecture
- Limited benefits compared to full migration

## Implementation Plan

### Phase 1: Foundation (Weeks 1-2)

#### 1.1 Create KubernetesJS Templating System
```typescript
// New file: clients/js/packages/client/src/kubernetes/templating.ts
export class KubernetesTemplatingEngine {
constructor(private config: StarshipConfig) {}

generateDeployments(): V1Deployment[] {
return this.config.chains.map(chain => this.createChainDeployment(chain));
}

generateServices(): V1Service[] {
return this.config.chains.map(chain => this.createChainService(chain));
}

generateConfigMaps(): V1ConfigMap[] {
return this.config.chains.map(chain => this.createChainConfigMap(chain));
}
}
```

#### 1.2 Add YAML Generation Capability
```typescript
// New file: clients/js/packages/client/src/kubernetes/yaml-generator.ts
export class YAMLGenerator {
constructor(private engine: KubernetesTemplatingEngine) {}

async generateAllYAML(outputDir: string): Promise {
const resources = this.engine.getAllResources();
await this.writeYAMLFiles(resources, outputDir);
}
}
```

#### 1.3 Update Client Interface
```typescript
// Modified: clients/js/packages/client/src/client.ts
export class StarshipClient {
// Add new methods
async generateYAML(outputDir: string): Promise {
const generator = new YAMLGenerator(this.templatingEngine);
await generator.generateAllYAML(outputDir);
}

async deployFromYAML(yamlDir: string): Promise {
const client = new KubernetesClient({ /* config */ });
// Deploy resources using KubernetesJS
}
}
```

### Phase 2: KubernetesJS Integration (Weeks 3-4)

#### 2.1 Add KubernetesJS Client
```typescript
// New file: clients/js/packages/client/src/kubernetes/client.ts
import { KubernetesClient } from 'kubernetesjs';

export class StarshipKubernetesClient {
private client: KubernetesClient;

constructor(config: StarshipClientConfig) {
this.client = new KubernetesClient({
restEndpoint: config.kubernetesEndpoint || 'http://127.0.0.1:8001'
});
}

async deployChain(chain: Chain): Promise {
const deployment = this.createDeployment(chain);
await this.client.createAppsV1NamespacedDeployment({
path: { namespace: this.namespace },
body: deployment
});
}
}
```

#### 2.2 Replace Shell Commands
```typescript
// Replace shell-based operations
// Before:
this.exec(['kubectl', 'get', 'pods', ...this.getArgs()]);

// After:
const pods = await this.k8sClient.listCoreV1NamespacedPod({
path: { namespace: this.namespace }
});
```

#### 2.3 Update Dependency Management
```typescript
// Modified: clients/js/packages/client/src/deps.ts
export const dependencies: Dependency[] = [
{
name: 'kubectl',
url: 'https://kubernetes.io/docs/tasks/tools/',
installed: !!shell.which('kubectl'),
required: false // Optional for API access only
},
{
name: 'docker',
url: 'https://docs.docker.com/get-docker/',
installed: !!shell.which('docker')
}
// Remove helm dependency
];
```

### Phase 3: Advanced Features (Weeks 5-6)

#### 3.1 Resource Monitoring
```typescript
// New file: clients/js/packages/client/src/kubernetes/monitoring.ts
export class ResourceMonitor {
async waitForPodsReady(namespace: string, timeout: number): Promise {
const watch = this.client.watchCoreV1NamespacedPod({
path: { namespace },
query: { watch: true }
});

// Implement pod readiness watching
}
}
```

#### 3.2 Port Forwarding via API
```typescript
// Replace kubectl port-forward with API calls
export class PortForwardManager {
async forwardPort(podName: string, localPort: number, remotePort: number): Promise {
// Use KubernetesJS port forwarding API
}
}
```

## Migration Benefits

### Immediate Benefits
1. **Better Error Handling**: Structured error responses from Kubernetes API
2. **Improved Debugging**: Direct access to resource status and events
3. **Platform Independence**: No shell command dependencies
4. **YAML Inspection**: Users can review generated resources before deployment

### Long-term Benefits
1. **Enhanced Testing**: Mock KubernetesJS client instead of shell commands
2. **Advanced Features**: Real-time resource monitoring, streaming logs
3. **Better Integration**: Easier to integrate with CI/CD and other tools
4. **Reduced Dependencies**: Eliminate Helm requirement

## Breaking Changes

### Configuration Changes
- New optional `kubernetes.endpoint` configuration
- Deprecated Helm-specific settings
- New YAML output directory settings

### CLI Changes
```bash
# New commands
starship generate-yaml --config config.yaml --output ./k8s-yaml
starship deploy-yaml --yaml-dir ./k8s-yaml
starship deploy --use-k8s-client --config config.yaml

# Modified existing commands (backward compatible)
starship deploy --config config.yaml # Still works with Helm
```

### API Changes
- New `generateYAML()` method on StarshipClient
- New `deployFromYAML()` method
- Deprecated shell-based methods (with warnings)

## Testing Strategy

### Unit Tests
- Mock KubernetesJS client for testing
- Test YAML generation logic
- Test resource creation logic

### Integration Tests
```typescript
// Example test using provided KubernetesJS patterns
describe('Starship KubernetesJS Integration', () => {
it('should deploy PostgreSQL using KubernetesJS', async () => {
const client = new StarshipKubernetesClient(config);
await client.deployChain(postgresChain);

const deployment = await client.getDeployment('postgres-pgvector');
expect(deployment.status?.readyReplicas).toBe(1);
});
});
```

### E2E Tests
- Full deployment cycle using KubernetesJS
- Comparison tests: Helm vs KubernetesJS output
- Performance benchmarks

## Documentation Updates

### User Guide Updates
1. New "YAML Generation" section
2. Updated installation guide (remove Helm requirement)
3. New troubleshooting guide for KubernetesJS-specific issues
4. Migration guide from Helm to KubernetesJS

### Developer Documentation
1. KubernetesJS architecture overview
2. Templating system documentation
3. Contributing guide updates
4. API reference for new methods

## Dependencies

### New Dependencies
```json
{
"kubernetesjs": "^latest",
"js-yaml": "^4.1.0" // Already present
}
```

### Removed Dependencies
- Helm (clients no longer need it installed)
- Reduced kubectl dependency (optional for advanced use cases)

## Success Criteria

### Phase 1 Success Criteria
- [ ] YAML generation produces valid Kubernetes manifests
- [ ] Generated YAML matches current Helm output (functional equivalence)
- [ ] No breaking changes to existing CLI commands
- [ ] Comprehensive test coverage for templating engine

### Phase 2 Success Criteria
- [ ] KubernetesJS deployment works end-to-end
- [ ] All shell commands replaced with API calls
- [ ] Error handling improvements measurable
- [ ] Performance parity or improvement vs Helm

### Final Success Criteria
- [ ] Zero shell dependencies for core functionality
- [ ] User can inspect YAML before deployment
- [ ] Improved error messages and debugging experience
- [ ] Smooth migration path for existing users
- [ ] Documentation fully updated

## Risk Mitigation

### Technical Risks
1. **KubernetesJS API limitations**: Mitigation - Prototype critical paths early
2. **Performance regression**: Mitigation - Benchmark each phase
3. **Feature parity gaps**: Mitigation - Comprehensive feature mapping exercise

### User Experience Risks
1. **Breaking changes**: Mitigation - Maintain backward compatibility during transition
2. **Learning curve**: Mitigation - Comprehensive documentation and examples
3. **Migration complexity**: Mitigation - Automated migration tools where possible

## Related Issues

- [ ] #XXX - Add YAML generation capability
- [ ] #XXX - Implement KubernetesJS client integration
- [ ] #XXX - Replace shell commands with API calls
- [ ] #XXX - Update documentation for Starship 2.0
- [ ] #XXX - Create migration guide from Helm to KubernetesJS

## Timeline

- **Week 1-2**: Phase 1 - YAML Generation Foundation
- **Week 3-4**: Phase 2 - KubernetesJS Integration
- **Week 5-6**: Phase 3 - Advanced Features & Polish
- **Week 7**: Testing, Documentation, Release Preparation
- **Week 8**: Release and User Migration Support

---

This epic represents a fundamental architectural improvement that will make Starship more maintainable, debuggable, and powerful while providing better user experience through YAML inspection and improved error handling.