Important: Jan 21 Sunday PODs status check - Reminder

Title of ticket:

Once the outage is complete and traffic resumes, you should check on the status of your application. Most likely, your application will continue to function as expected, and no action will be required from your team. If your application pods did restart during the outage for any reason and have failed spin up again correctly, you should be able to simply restart the pods once again and, now that internet access has been restored, the pods should be able to start correctly without any further problems.

#### Description
What's happening?

On Sunday, January 21st, 2024, the core switches servicing the Kamloops data center will be replaced. This will cause a network outage to the entire data center that may last up to 1 hour, starting at 2am.

Many of you may already be aware of this event, as a Technical Bulletin has already been released by the OCIO. If you didn't recieve notification of the technical bulletin, you can find more information on Change Management Ticket KB0033041

What does this mean for my OpenShift application?

This change only affects network access to the data center. This means that the OpenShift clusters in the Kamloops data center will not restart or otherwise cause pods to restart. The clusters themselves will continue to function as normal during this outage window - it's just that there will be no incoming or outgoing traffic to or from any of the Kamloops clusters at this time.

The Gold, Silver, and Emerald clusters will be inaccessible during this time. Gold-DR will be unaffected, due to its location in the Calgary data center.

My application is on Silver or Emerald: what should I do to prepare?

You should prepare for your application to be inaccessible for up to 1 hour in the early morning on Sunday, January 21st, 2024. Nothing about this outage will cause your application pods to restart or redeploy, so it's unlikely that this will have a negative impact on your application - traffic should simply resume once the switches are in place. From your application's perspective, it will be as if there was simply no traffic during the outage window.

However, if your application pods restart during this window, they may have problems starting up again because they won't have access to many core services provided either by the Platform Team or by external vendors. In particular, please be aware that your application pods will not have access to either Artifactory or Vault during this time, because both of these applications will fail over to Gold-DR during the outage window. If your application is dependent on either of these services and your application restarts during the outage, it's likely that your pod will crashloop because it won't have access to its images or secrets during this time.

**Once the outage is complete and traffic resumes, you should check on the status of your application. Most likely, your application will continue to function as expected, and no action will be required from your team. If your application pods did restart during the outage for any reason and have failed spin up again correctly, you should be able to simply restart the pods once again and, now that internet access has been restored, the pods should be able to start correctly without any further problems.**

Your team should consider making sure that your application can gracefully handle a lack of internet access for up to 1 hour. If your liveness or readiness checks fail without internet access, this will cause your pods to restart during the outage window. Your application is likely to be most stable during this outage if the pods don't restart, so it might be a good idea for your team to try to fix dependencies like this, if possible.

My application is on Gold: what should I do to prepare?

If your application has automatic fail-over set up between Gold and Gold-DR, you aren't required to take any action at all. Your application should fail-over to Gold-DR once the outage begins, and will fail back once the outage is over.

Your team may want to test your automatic fail-over process before the outage window to ensure that everything works as expected, both during the fail-over to Gold-DR and the recovery back to Gold.

Some teams plan to manually fail-over certain services from Gold to Gold-DR during the week before the January 21st outage window, and will manually fail during the week following. The Platform Team intends to take this approach with our core services in order to guarantee service availability during the outage window. Your team may want to take this approach if your failover process requires any manual intervention, or if you wish to have your technical team available to fix any issues during failover.

I have questions!

As always, feel free to contact the Platform Team at PlatformServicesTeam@gov.bc.ca or contact us on RocketChat if you have any questions about how this change impacts the Private Cloud Platform and/or any of our related services.

Because the actual change is being performed by the OCIO and not the Platform Team, if you have questions about the details of the change itself, please contact the relevant team via Change Management Ticket KB0033041

Love,

Your Platform Team

#### Dependencies
Are there any dependencies?

#### DOD
- [ ] List the items that need to be complete for this ticket to be considered done
- [ ]
- [ ]
- [ ]
- [ ]

Start by reading issue #4872 and confirm whether this outage reminder is still intended to be maintained in the repository. The issue provides no file, test, or concrete change and its DOD remains a placeholder, so clarify the intended documentation update and acceptance criteria before beginning.

Written by the indexing model from the issue text.