environment promotion strategy

pybot/ pyback:

1. Work locally in development, can target a test slack
2. Push to staging branch, targets a test slack but shows what happens when deployed in ECS.
3. Push to master, targets prod

@OperationCode/front-end and current frontend:

1. Work locally to target whatever we set in environment variables (staging, prod)
2. Create pull request, which on merge promotes the production to this

Backend does the same as front-end, both front-end and backend you can go into kubectl and manually target whatever container tag you'd like. When I request the current images for all pods, we'll see some oddities:

C:\Users\wimo7\Desktop> kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr [:space:] '\n'
operationcode/operationcode_frontend:latest
operationcode/operationcode_frontend:latest
... supporting containers...
operationcode/operationcode_backend:1245
operationcode/operationcode_backend:latest
operationcode/operationcode_backend:1245
operationcode/operationcode_backend:latest
operationcode/operationcode_frontend:2327
operationcode/operationcode_backend:1269
operationcode/operationcode_backend:1269
operationcode/operationcode_backend:1269
operationcode/operationcode_backend:1269
operationcode/operationcode_frontend:2334
operationcode/operationcode_frontend:2334
bitnami/redis:3.2.9-r2
operationcode/town-crier:latest

In the above case, we see that frontend is using both 2334and latest. while backend is using 1269, 1245 and latest.

I don't know the correct command to list the namespace for each of these. But I remember find out that we are setting staging cube as "latest" and the prod as a numerical value. I don't know why we have 3 different backend image versions.

After talking with @kylemh (IIRC) he doesn't need a way to promote a deployed code instance to staging. For their purposes it's fine to have staging only running locally. But I think going to a strategy where we promote to staging on merge to master, by tags, and then manually promote by adding a production tag would be more ideal, or automatically based on some criteria.

Our initial concern was having k instances of staging for k developers. But I don't see that as a current problem.

Lastly, we need a way that data sources (backend, pyback) RDS instances, have data in production that propagate down so that when we update those instances we can easily carry that over. I think for the time being a nightly copy over, plus ability to manually trigger, would be fine. With the knowledge that if you are working on something that requires database migrations you'll have to perform the migration again with the updated data.

Start by reviewing the staging and master deployment paths, the ECS and kubectl image behavior, and the RDS data-source process described in the issue. Inventory how frontend and backend images are currently tagged and deployed, including the differing versions shown. Done would require an agreed promotion strategy and a defined, repeatable approach for copying production data to lower environments.