Setting up TLS on Load Balancers

Migration guide for glb users without TLS

You will need to remove the ingress section from the gap.yaml. For example:

    ingress:
      enabled: true # set to false
      class: gce #remove
      backendConfigName: <application-name>-backendconfig #remove
      annotations: #remove
        nginx.ingress.kubernetes.io/server-snippet: "location /metrics {\n  deny all;\n}" #remove

Remove your gap_backendconfig.yaml file from the gap folder.
Follow the guide for Setting up a new Load Balancer with TLS below.
Remember to click prune when syncing on ArgoCD.

Setting up a new Load Balancer with TLS

gap.yaml

You will need to make the following changes to your deployment in the gap.yaml:

Set the ingress.enabled to false.
Add podAnnotations like in the below example, don’t forget to change .

name: <application-name>
namespace: <your-namespace>
deployments:
  <deployment-name>:
    podAnnotations:
      sidecar.istio.io/userVolume: '{"tls-secret":{"secret":{"secretName":"<application-name>-self-signed-tls"}}}'
      sidecar.istio.io/userVolumeMount: '{"tls-secret":{"mountPath":"/etc/istio/tls-certs/","readOnly":true}}'
    ingress:
      enabled: false

gap_glb_tls_bundle.yaml

This is an example configuration showing how to use a Google Cloud Load Balancer ingress (class gce) that is connecting to our GAP application with HTTPS. GCP Load Balancers normally use HTTP so we have to tell GCP to use HTTPS (see the Service resource). Because GLB lives outside of the kubernetes cluster we cannot just use service mesh auto mTLS for it, we’ll show how to configure istio-proxy to serve a custom self-signed certificate on a specific port that will be able to serve the Load Balancer. We can use a self-signed certificate here because we have both sides under our control and GLB does not verify it.

You need to put the below yaml file in your GAP folder and replace the following:

<application-name>
<same-as-previous-glb-ingress-name> ⚠️ ⚠️ ⚠️ It is important to keep the same previous GLB ingress name for the below custom one. Otherwise it is a destructive action, a new GLB would be created with a new IP, and an A record change on our side would be needed which would cause downtime on production.
<your-namespace>
<hostname>: the fully qualified domain name
- for staging: <application-name>-staging.gservice.emarsys.com
- for production: <application-name>.gservice.emarsys.net

# Creates the self-signed certificate in the specified secret
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: <application-name>
  namespace: <your-namespace>
spec:
  secretName: <application-name>-self-signed-tls
  commonName: <application-name>
  issuerRef:
    name: <application-name>-issuer
    kind: Issuer
    group: cert-manager.io
---
# Self-signed certificate issuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: <application-name>-issuer
  namespace: <your-namespace>
spec:
  selfSigned: {}
---
# This configures istio on your pod to serve the mounted self-signed certificate on port 8443 instead of the normal auto-mtls. 
apiVersion: networking.istio.io/v1
kind: Sidecar
metadata:
  name: <application-name>-sidecar
  namespace: <your-namespace>
spec:
  workloadSelector:
    labels:
      app: <application-name>
  ingress:
    - port:
        number: 8443
        protocol: HTTPS
        name: external
      defaultEndpoint: 127.0.0.1:8080
      tls:
        mode: SIMPLE
        privateKey: "/etc/istio/tls-certs/tls.key"
        serverCertificate: "/etc/istio/tls-certs/tls.crt"
        caCertificates: "/etc/istio/tls-certs/ca.crt"
    - port:
        number: 8080
        protocol: HTTP
        name: internal
      defaultEndpoint: 127.0.0.1:8080
---
# Ingresses target Service objects, which target Pods. You have to create a Service definition manifest that targets your pods with the annotation:
apiVersion: v1
kind: Service
metadata:
  name: <application-name>
  namespace: <your-namespace>
  annotations:
  # The _cloud.google.com/neg_ annotation instructs the controller to create and manage a Network Endpoint Group (NEG) which is required for container-native load balancing.
  # NEG Services are regular Kubernetes Services, which allows Google Load balancers to connect to the pods directly.
    cloud.google.com/neg: '{"ingress": true}'
    cloud.google.com/backend-config: '{"default":"<application-name>"}'
    service.alpha.kubernetes.io/app-protocols: '{"https":"HTTPS","http":"HTTP"}'
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8080
    - port: 443
      name: https
      targetPort: 8443
  selector:
    app: <application-name>
  type: ClusterIP
---
# FrontendConfigs are referenced in the Ingress object with the `networking.gke.io/v1beta1.FrontendConfig: <application-name>` annotation
# The FrontendConfig CRD references our gap SSL Policy.
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: <application-name>
  namespace: <your-namespace>
spec:
  sslPolicy: gap-ssl-policy
---
# The ingress class annotation does the following:
# instructs the default ingress controller to _ignore_ this Ingress
# instructs the GCE controller to pick up this ingress and start provisioning the GLB resources

# The cert-manager annotation is for ensuring compatibility with the GLB. Normally, cert-manager creates a separate Ingress object for the challenge URL. This annotation instructs it to put the challenge URI in the same Ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: gce
    acme.cert-manager.io/http01-edit-in-place: "true"
    networking.gke.io/v1beta1.FrontendConfig: <application-name>
  name: <same-as-previous-glb-ingress-name>
  namespace: <your-namespace>
spec:
  rules:
    - host: <hostname>
      http:
        paths:
          - backend:
              service:
                name: <application-name>
                port:
                  number: 443
            path: /
            pathType: Prefix
  tls:
    - hosts:
        - <hostname>
      secretName: <application-name>-tls
---
# This policy enables strict mTLS for all workloads with label app: <application-name>, excluding ports 8443 and 15090. 15090 is used by istio proxy metrics, and 8443 is the one GLB backends are connecting to as configured in the Sidecar resource.
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: <application-name>-strict-mtls
  namespace: <your-namespace>
spec:
  selector:
    matchLabels:
      app: <application-name>
  mtls:
    mode: STRICT
  portLevelMtls:
    "8443":
      mode: DISABLE
    "15090":
      mode: DISABLE
---
# Ensure that calls to the healthcheck path is allowed
# Google Load Balancer backends are checking the pods directly
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: <application-name>-web
  namespace: <your-namespace>
spec:
  action: ALLOW
  rules:
  - to:
    - operation:
        methods:
        - GET
        paths:
          # this must match the healthcheck path in BackendConfig shown below
        - /healtcheck
    (...)
---
# BackendConfigs are referenced by a Service object with the cloud.google.com/backend-config: '{"default":"<application-name>"}' annotation
# The BackendConfig CRD specifies custom settings for the corresponding backend service's health check.
# These are just example values you can read more about custom health check configuration in the Further resources section below. These values depend on your specific use-case.
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: <application-name>
  namespace: <your-namespace>
spec:
  healthCheck:
    checkIntervalSec: 5
    timeoutSec: 5
    port: 8443
    type: HTTPS
    requestPath: /healthcheck

Further resources

In case you are interested you can read more about ingress configuration here. Here is the documentation for custom health check configuration using BackendConfig.