Google Load Balancer switchover from Nginx

These are the steps that we have to do as prep before the switchover can happen. It is a good idea to do these things a day before.

1. Reduce CNAME TTL to 5 minutes (DNS Made Easy)
2. Deploy the GLB resources in parallel while keeping the old setup.

   It took close to 40 minutes for the google resources to get ready once, so don’t panic if the resources are taking their time.

3. Check that things still work on staging with the mostly dormant GLB resources being deployed and that the new resources are also healthy. Make sure the certificates are ready.
4. Create an A record with the ingress IP and a 30 or 60 sec TTL, but DO NOT use it yet. (DNS Made Easy)

Do these steps together with the team who needs the switchover.

0. Obviously do these at staging first and only think about production if everything went smoothly.
1. Double Check that all the prep work was done and enough time has passed so the old TTL can’t be a problem.
2. Set the TTL to even lower to make sure changing back could happen quickly. 30 or 60 seconds and wait 5 minutes before step 3 (DNS Made Easy)
3. After waiting a bit change the CNAME’s alias to field to the name of the A record with the new ingress IP. (DNS Made Easy)
4. Confirm the new setup works, everything should use it in 1-2 minutes.
5. Monitor for a while, make sure no strange behaviour and new 4XX 5XX errors happen due to the change.
6. When feeling confident, Nginx can be cleaned up. (Should be at least a day after the switchover)
7. Increase the TTL when we are happy with the new system and a rollback is very unlikely.

If major issues are detected after the switchover.

1. Revert the change to the CNAME.
2. Verify traffic is going through the old setup. Should not take more than 2 minutes for it to change.
3. Figure out the possible GLB config issues.