SPIKE: Investigate executing rspm commands in PPM container for Git Builders setup

Problem

After mounting SSH keys into the PPM container (as described in #2381), we need to import these keys into PPM using the rspm import ssh-key command. This requires executing commands within the running PPM container programmatically from the team-operator controller. However, managing idempotency and workflow orchestration presents significant challenges.

Investigation Goals

Research and evaluate approaches for executing commands in the PPM container with proper idempotency and workflow management.

Key Questions to Answer

1. How do we ensure idempotency when importing SSH keys (avoid re-importing unchanged keys)?
2. How do we handle state management and track which keys have been imported?
3. Should we use a lightweight configuration management tool instead of direct command execution?
4. How do we handle key rotation and updates without disrupting service?
5. What's the best approach for error handling and retry logic?

Technical Approaches to Evaluate

1. Direct command execution via remotecommand (https://pkg.go.dev/k8s.io/client-go/tools/remotecommand)

   • Pros/cons of managing state ourselves
   • Complexity of implementing idempotency

2. Lightweight configuration management tools

   • Evaluate tools that could handle the workflow/idempotency problem
   • Consider overhead and complexity of adding another tool

3. Alternative approaches

   • Init containers
   • Sidecar containers
   • Operator-managed ConfigMaps for state tracking

Success Criteria

• Document recommended approach with justification
• Prototype proof-of-concept if feasible
• Identify risks and mitigation strategies
• Estimate implementation effort

Dependencies

This spike depends on understanding the SSH key mounting infrastructure from #2381.

Migrated from rstudio/ptd#2382

Start by reviewing the SSH key mounting infrastructure in #2381 and the Kubernetes remotecommand package. Compare direct execution, configuration-management, init/sidecar, and ConfigMap approaches for idempotency, state, rotation, retries, and workflow management; done means documenting a recommendation, risks, implementation estimate, and a proof of concept if feasible.