aws-samples / aws-samples/appmod-blueprints

Port platform GitOps/chart fixes from feature/cloudfront-on-agent-platform to main

Open
#747 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
105
Forks
62
Avg merge
11h 17m
Merged PRs (30d)
76

Description

## Summary

While operating a `hub` cluster deployed from `feature/cloudfront-on-agent-platform`, we found and fixed several GitOps/chart bugs. These fixes are committed on that feature branch and should be ported to `main` (verifying each against `main`'s current state, since some depend on environment conventions).

Cluster context: EKS Auto Mode; ArgoCD/Kro/ACK via EKS Capabilities; CloudFront exposure mode.

## Fixes to port

### 1. GitLab: add `startupProbe` so the pod survives first boot
- **File:** `gitops/addons/charts/gitlab/templates/gitlab.yaml`
- **Problem:** liveness/readiness used `initialDelaySeconds: 180` with no startup probe; GitLab Omnibus was killed ~4.5 min into its first `reconfigure` + DB migrations -> crash loop (250+ restarts), Deployment "exceeded progress deadline", app Degraded.
- **Fix:** add a `startupProbe` (`/-/readiness`, `periodSeconds: 15`, `failureThreshold: 60` ~= 15 min budget) and relax liveness/readiness (`failureThreshold: 5`, drop `initialDelaySeconds`).
- **Commit:** `25771f3e`

### 2. Keycloak: harden `add_client_scope` so one failed client doesn't abort the whole config job
- **File:** `gitops/addons/charts/keycloak/templates/keycloak-config.yaml`
- **Problem:** the PostSync `config` job runs `set -ex -o pipefail` and `jq -e`. When a client (e.g. `kargo`/`gitlab`) failed to create (invalid `rootUrl` from an empty domain), `add_client_scope` did `CLIENT_ID=$(keycloak_get ... | jq -e ...)` -> empty -> non-zero -> whole script aborts -> job hits backoffLimit -> keycloak app stuck Progressing forever.
- **Fix:** tolerate a missing client (`|| true` + skip/`return 0` when `CLIENT_ID` empty) so the job completes and only the affected client is skipped.
- **Commit:** `25771f3e`

### 3. gitlab-init: create repos under the workshop user's namespace, not `root`
- **File:** `gitops/addons/charts/gitlab/templates/init-job.yaml`
- **Problem:** the init job created `fleet-config` and the application repos with the **root** PAT via `POST /api/v4/projects/` (no namespace) -> repos landed under `root/`. The script then cloned/pushed `$GIT_USERNAME/$WORKING_REPO` (e.g. `user1/fleet-config`), which 404'd -> `set -e` aborted -> `BackoffLimitExceeded`.
- **Fix:** resolve the target user id and create via `POST /api/v4/projects/user/:id`; make existence/id lookups match `path_with_namespace == "$GIT_USERNAME/"`.
- **Commit:** `8bc9de8a`

### 4. KubeVela: reference the existing `default` AWS ProviderConfig
- **Files:** `gitops/addons/charts/kubevela/templates/components/{service-account,s3-bucket,rds-cluster,ddbtable-component}.yaml`, `.../traits/component-policy.yaml`
- **Problem:** component/trait CUE hardcoded `providerConfigRef: name: "provider-aws-config"`, which does not exist (the cluster only has `default`, PodIdentity-based). Crossplane managed resources (e.g. `devlake-mysql-access-iam-role`) couldn't connect -> no `RoleArn` -> dependent `PodIdentityAssociation` failed with `CannotResolveResourceReferences`.
- **Fix:** point all KubeVela AWS components/traits at `default`.
- **Note for main:** verify whether `main` expects a ProviderConfig named `provider-aws-config` (and creates one) or also uses `default`. Align to whatever actually exists.
- **Commit:** `49fc8aeb`

### 5. DevLake: drop `ttlStrategy` on `mysql-setup-workflow`
- **File:** `gitops/addons/charts/devlake/templates/setup-workflow.yaml`
- **Problem:** the Workflow is a sync-wave `-1` ArgoCD-tracked resource but had a `ttlStrategy`, so the workflow-controller deleted it ~2 min after success. ArgoCD then saw it `Missing` -> re-applied -> it ran again -> deleted again (perpetual loop), leaving `devlake-hub` `OutOfSync/Missing` and the operation stuck "waiting for healthy state of Workflow/mysql-setup-workflow".
- **Fix:** remove `ttlStrategy` so the Workflow persists as `Succeeded` (Healthy/Synced). (Alternative: convert to an ArgoCD hook with `hook-delete-policy`, but that needs the SA/ClusterRoleBinding to become hooks too for ordering.)
- **Commit:** `1b9a5493`

### 6. KubeVela: broaden `ignoreDifferences` for vela-core admission webhooks
- **File:** `gitops/addons/registry/platform.yaml`
- **Problem:** `kubevela-hub` stayed `OutOfSync` even though `caBundle` was ignored. The real drift was in fields the API server/vela-core default or mutate: `failurePolicy` (chart `Ignore` -> live `Fail`), `timeoutSeconds`, `matchPolicy`, `reinvocationPolicy`, `namespaceSelector`, `objectSelector`, `rules[].scope`. Cosmetic drift makes it hard to spot real drift.
- **Fix:** replace the indexed `caBundle`-only `jsonPointers` with `jqPathExpressions` (`.webhooks[].`) covering all these fields across any number of webhooks, for both Mutating/Validating webhook configs.
- **Commit:** `ba823514`

## Notes
- Commits above are on `feature/cloudfront-on-agent-platform`.
- Fixes #1-#3, #5, #6 are environment-agnostic. Fix #4 needs a quick check of `main`'s ProviderConfig naming convention before porting.
- On the live cluster, several of these also required clearing stuck ArgoCD sync operations (patch `operation`/`status.operationState` to null + hard refresh) and forcing a KubeVela re-render (`app.oam.dev/publishVersion`) - operational steps, not code changes.

Contributor guide

Open the contributing guide

Research direction

Compare commits 25771f3e, 8bc9de8a, 49fc8aeb, 1b9a5493, and ba823514 with main, starting with the listed GitLab, Keycloak, KubeVela, DevLake, and platform chart files. Check main's AWS ProviderConfig naming before porting the KubeVela change; done means all six fixes are aligned with main's current conventions and the affected manifests and jobs reflect the described behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kubernetes, shell
Domain
ci-cd, cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.