Ops guide

Note: every change made to the cluster directly (e.g. modifying the number of replicas in the YAML via the web console, or other tool) should be captured in your configuration files in the repository of your application, because they are only staying in effect until the next deployment.

Interacting with the cluster from terminal

# connect to the target cluster (production, staging)
gcloud container clusters get-credentials gap-production --region europe-west3 --project ems-gap-production
gcloud container clusters get-credentials gap-staging --region europe-west3 --project ems-gap-stage

# after the first connection you can change between cluster contexts
kubectl config get-contexts
kubectl config use-context <context_name>

# restart a single deployment
kubectl rollout restart deployments/[your-pod] -n [your-namespace]
# k9s > type :dp > move cursor over the deployment to be restarted > press <ctrl> + t
# if the command returns "error: unknown command "restart deployment ...", you need to upgrade your kubectl version

# restart all deployments (web,worker, etc) of your application, needs kubectl v1.15
namespace='your-namespace'
application_name="application-name"
deploys=(`kubectl get deployment -l applicationName=$application_name -n $namespace | tail -n+2 | cut -d ' ' -f 1`)
for deployment in "${deploys[@]}"
do
    kubectl rollout restart deployments/$deployment -n $namespace
done

# check your deployment's status
kubectl rollout status -n [your-namespace] deployment/[APPLICATION_NAME]

# run a one-off pod in the cluster
kubectl run \
--overrides='{
  "spec": {
    "securityContext": {
      "runAsUser": 1000,
      "fsGroup": 1000
    },
}' --generator=run-pod/v1 -it --rm --image [image-to-run] -n [your-namespace] [pod-name] -- /bin/bash

# open a shell into a pod
kubectl exec -it [pod_name] --namespace=[your-namespace] bash

# scale your application horizontally
kubectl scale deployments/[deployment_name] --replicas=[replica_number] -n [your-namespace]

How to run a one-shot container including your applications environment

Prepare a yaml file from the template below, modifying according to your application’s name and image and namespace. Save this file as one-shot.yaml

apiVersion: v1
kind: Pod
metadata:
  name: <your-application-name>-one-shot
  namespace: your-namespace
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
spec:
  containers:
    - envFrom:
        - secretRef:
            name: <your-application-name>
      image: eu.gcr.io/ems-gap-images/<your-app-name>:latest
      name: <your-application-name>-one-shot
  restartPolicy: Never
  securityContext:
    runAsUser: 1000

# Determine the image for your yaml
kubectl get deployments/<your-application-name> -n your-namespace -o json | jq ".spec.template.spec.containers[0].image"

# create the stand alone pod
kubectl apply -f one-shot.yaml -n your-namespace

Run command in this new pod.

See How to run command in your existing pod

# delete your pod after you finished your work
kubectl delete -f one-shot.yaml -n your-namespace

How to run a command in your existing pod

# get one of your running pods exact name
kubectl get pod -l applicationName=your-applicationName -n your-namespace | grep web

# exec into the above defined pod and execute your command
kubectl exec -n your-namespace your-pod-name --  sh -c "your command to run"

How to access endpoints of meshed services

Services that are meshed and are without ingress can be locally tested by forwarding the pods’ ports to your machines. This method will expose the desired service endpoints to interact with during local testing and development. With this workaround, it is advised to remove ingress if not neccessary.

kubectl port-forward -n <ns> <podname> <port>

For a more intuitive option k9s also can do the trick. By selecting your pod and pressing shift+f, the selected endpoint will be available on your localhost. Either pressing shift+f again or closing k9s will terminate the portforwarding.

How to find out the external IPs of the pods of your deployment

deploymentName="<your-deployment-name>" # eg.: gap-docs-web
namespace="<your-namespace>" # eg.: cloud-platform
nodes=`kubectl get pods -n $namespace -l app=$deploymentName -o=jsonpath='{.items[*].spec.nodeName}'`
for i in `echo $nodes | tr -s ' ' '\n'`; do
    echo `kubectl get node -o=jsonpath='{.status.addresses[?(@.type=="ExternalIP")].address}' $i`;
done

How to find unmanaged resources

It is possible that sometimes you forget to clean up resources after having it removed from the gap folder when syncing argocd without prune or when resources that existed before argocd are not deployed anymore but were left on the cluster. Argo CD adds the argocd.argoproj.io/instance label to all the kubernetes resources it is managing. Using this knowledge it is straightforward to query any of the resources that are not managed by Argo CD.

kubectl get svc,deployment,configmap,serviceaccount,ingress,cronjob,statefulset -n <namespace> -l "!argocd.argoproj.io/instance"

Keep it in mind that this only works for directly managed resources and not for derived resources like Pods or for externally managed like Secrets (gap-cli).

Deployment

How to rollback quickly

Go to Argo CD and search for your application, open it.

On the History and rollback panel search for the commit hash to roll back to and click Rollback in the top-right menu.

Use the hash in the Authored By section, and NOT the Revision

More info on the Argo CD page

How to interact with your cluster through the UI

How to run your scheduled job once

Find your job here
Click Run now

How to suspend your scheduled job

Find your job here
Click Suspend

How to scale replicas

Find your deployment here
Select Action > Scale

Note: this will only apply until your next deployment.

How to scale memory and cpu your application

Find your deployment here
Select Edit, and edit the corresponding sections (spec:template:spec:containers:resources) in your configuration.

Note: this will only apply until your next deployment.

Configuration, environment, secrets

# edit your application's configuration
gap-cli config:edit -a [APPLICATION_NAME] -n [your-namespace]

Database

How to connect to PostgreSQL db

See Connecting to Cloud SQL

MemoryStore (Redis)

### Where to change the configuration of your MemoryStore instance

Every application should have their own Google projects (staging and production) where we can access our Redis instances through Memorystore module. Change the project name accordingly in the link example below:

https://console.cloud.google.com/memorystore/redis/instances?project=

Configuring database flags

gcloud config set project [PROJECT_NAME]
gcloud sql instances describe [INSTANCE_NAME]
gcloud sql instances patch [INSTANCE_NAME] --database-flags [FLAG1=VALUE1,FLAG2=VALUE2]

How to backup database

See the backup settings for your database (stage project in example) in its edit menu https://console.cloud.google.com/sql/instances/[your-database-name]/edit-performance-class?project=ems-gap-plugins-app-conn-s

Restore database without outage

Go here, select your application’s project. Create a new SQL instance.
Select the backup of your database to restore in its backup menu.
More actions > Restore > Select the new database
After the migration change the database url in the configuration of your application. See above in the Configuration section.