Refactor demo01 customizations to use new custom K8s resources pattern

Refactor demo01 customizations to use new custom K8s resources pattern

Context

PR rstudio/ptd#2501 introduced a new pattern for deploying custom Kubernetes resources through the custom_k8s_resources field in workload cluster configuration. This pattern provides a cleaner, more maintainable approach to deploying workload-specific resources.

Currently, demo01-staging has extensive customizations hardcoded in Python Pulumi code that should be refactored to use this new pattern.

Current State

The demo01 customizations are currently implemented in python-pulumi/src/ptd/pulumi_resources/aws_workload_clusters.py:815-994 via the _define_demo01_custom() method, which includes:

AWS IAM Resources (Cannot be migrated)

These will remain in Pulumi code as they are AWS-specific:

• OIDC Providers: Multiple OIDC providers including Okta, Keycloak, and Connect identity providers
• IAM Roles: Various service account roles for demo-s3, pharos (staging/prod), and sessions access

Kubernetes Resources (Can be migrated)

These could potentially be migrated to custom K8s resources:

• Service Accounts: Various service accounts with IRSA annotations
• RBAC: Kubernetes roles and role bindings for session management
• Any other Kubernetes manifests currently defined in the define_exec_sa_and_role() function

Migration Strategy

Phase 1: Analysis and Preparation

• Analyze which demo01 customizations can be moved to Kubernetes YAML files vs. must remain in Pulumi
• Identify all Kubernetes resources currently created by the demo01 customizations
• Create directory structure: infra/__work__/demo01-staging/custom_k8s_resources/

Phase 2: Create Custom K8s Resource Files

• Create YAML files for Kubernetes-native resources:
  • Service accounts with IRSA annotations
  • RBAC roles and role bindings
  • Any other K8s resources not requiring Pulumi logic
• Organize files with appropriate naming/ordering (e.g., 01-service-accounts.yaml, 02-rbac.yaml)

Phase 3: Update Configuration

• Add custom_k8s_resources field to infra/__work__/demo01-staging/ptd.yaml
• Configure appropriate subfolders for resource organization

Phase 4: Refactor Pulumi Code

• Remove Kubernetes resource creation from _define_demo01_custom() method
• Keep AWS-specific resources (OIDC providers, IAM roles) in Pulumi
• Clean up helper functions that are no longer needed

Phase 5: Testing and Validation

• Test deployment in demo01-staging environment
• Verify all resources are created correctly
• Confirm existing functionality is preserved

Benefits

1. Separation of Concerns: AWS resources stay in Pulumi, Kubernetes resources in YAML
2. Maintainability: Easier to modify K8s resources without touching Python code
3. Consistency: Follows the same pattern as other production workloads
4. Visibility: K8s resources are more visible in the file system
5. Version Control: Better diff tracking for resource changes

Acceptance Criteria

• All Kubernetes resources currently created by demo01 customizations are migrated to custom K8s resource files
• AWS IAM resources remain in Pulumi code
• ptd.yaml is updated with appropriate custom_k8s_resources configuration
• Demo01-staging deployment works correctly with the new pattern
• No functionality is lost during the migration
• Python code is cleaned up and simplified

Related

• Closes follow-up item from PR rstudio/ptd#2501
• Related to new custom K8s resources feature introduced in PR rstudio/ptd#2501
• Documentation: infra/CUSTOM_K8S_RESOURCES.md

Notes

This refactoring will make demo01 customizations more maintainable and align them with the new patterns established for other workloads. The migration should be done carefully to ensure no functionality is lost, especially around the complex OIDC and IAM integrations that demo01 requires for its various sites and integrations.

Start by reading _define_demo01_custom() in python-pulumi/src/ptd/pulumi_resources/aws_workload_clusters.py:815-994 and infra/CUSTOM_K8S_RESOURCES.md. Inventory the Kubernetes resources there, then inspect infra/work/demo01-staging/ptd.yaml and the related custom_k8s_resources pattern. Done means Kubernetes resources are migrated, AWS IAM resources remain in Pulumi, deployment works, and functionality is preserved.