Sending test logs

Overview

From the GAP applications the logs are first arrive at Google Stackdriver, from where they are forwarded to two PubSubs: one for the production applications and one for the staging applications. At the Emarsys side, these PubSubs are polled continuously by two Logstash instances.

In the docker ecosystem logging is done by printing messages to the standard output (stdout) and/or to the standard error (stderr). The document describes a workaround for sending logs to another process’ stdout/stderr.

Sending logs

This method uses the procfs (Wikipedia) inside the container to inject logs into the stdout file handle of another process. Procfs exposes kernel related data (including the file descriptors for stdout and stderr) for each processes.

A word of warning

The Cloud Platform team is not sure that the application’s stdout is locked when injecting data from another process. Although we haven’t heard of such case, in case of the application is logging at a high rate this might cause one or more applications log lines to become scrambled with the injected data.

Prerequisites

Application is already configured for LAAS
Logs are already arriving into LAAS (i.e. are visible in Kibana)
kubectl installed and configured for reaching the proper Kubernetes cluster (staging or prod)

Finding the pod for executing shell

The default replicas settings causes your deployments to run multiple pods (instances). In order to find one

# change to your team's namespace
namespace=tooling

# get a list of running pods
kubectl get pods -n ${namespace} | grep Running

# sample output
meeting-room-dashboard-clock-6ff5db4984-jk7gm                  1/1     Running     0          20h
meeting-room-dashboard-ssh-tunnel-79d5bf4b46-jfpnp             1/1     Running     0          4d20h
meeting-room-dashboard-web-77796fc54b-bfzgd                    1/1     Running     0          20h
meeting-room-dashboard-web-77796fc54b-srkqx                    1/1     Running     0          20h

Let’s choose meeting-room-dashboard-clock-6ff5db4984-jk7gm.

# execute a shell
kubectl exec -it meeting-room-dashboard-clock-6ff5db4984-jk7gm -n ${namespace} sh

Determine main application’s process id

In most cases, the main application in the container should have process ID 1. You can make sure with top or ps.

Using top:

$ top -c -b -n1
top - 09:50:16 up 5 days, 19:13,  0 users,  load average: 1.32, 1.26, 1.26
Tasks:   3 total,   1 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.1 us,  1.4 sy,  0.0 ni, 94.4 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 26752496 total, 10029500 free,  4694656 used, 12028340 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 21851152 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
      1 heroku    20   0  395792  30504  22432 S   0.0  0.1   0:15.90 php api/scripts/reservationUpdater.php forever
     38 heroku    20   0   18512   3484   2996 S   0.0  0.0   0:00.02 bash
     53 heroku    20   0   36492   2984   2628 R   0.0  0.0   0:00.00 top -c -b -n1

Using ps:

$ ps x
    PID TTY      STAT   TIME COMMAND
      1 ?        Ss     0:15 php api/scripts/reservationUpdater.php forever
     38 pts/0    Ss     0:00 bash
     54 pts/0    R+     0:00 ps x

Send logs from command line

For setting the pid value, refer to section 2.4 Determine main application’s process id

# change value accordingly
pid=1

echo "sample log to stdout" > /proc/${pid}/fd/1
echo "sample log to stderr" > /proc/${pid}/fd/2

Redirect script output to logs

For setting the pid value, refer to section 2.4 Determine main application’s process id

# change value accordingly
pid=1

sh script.sh 2>/proc/${pid}/fd/2 >/proc/${pid}/fd/1