How to set up Elastalert rules

Each team has their own elastalert rule repository where they can configure their own rules. To set up your repository and more information on how you can configure your rules please refer to this documentation.

If you are in the process of migrating your logs from Laas to Laas2, please refer to this document as well.

GAP application indexes

GAP application logs are being placed in the index named gap-${appName}.

How to distinguish staging from production

You can use the @gap.cluster_name field, its value can be gap-staging or gap-production.

Message field

If your log is in text format the field message will contain it. If your log is in json format the field message will contain your logged json’s message property.

Every field you’ve logged should be visible on the root level of the log.

Router type log based alerts

Every router log will contain amongst others the followings:

“router_log”: true # this way you can identify these logs
“router.host”: <your-app-name>-<your-deployment-name>.<your-namespace> # the component, which the log is referring to

Alert for status code

Please use the router.status field.

Alert for response time, common cases

Please use the router.request_time field, the value is in seconds.

Request timeout

You can use nginx HTTP status 504 in the application router logs.

Criteria to look for:

index: gap-app-name (replace with the proper index)
applicationName: ingress-nginx
router.status: 504

Example partial Elastalert rule:

name: test-app Request Timeout
description: test-app one or more request timeouts happened within an hour

index: gap-test-app # replace this with your application's index

type: frequency
num_events: 1
timeframe:
  hours: 1

filter:
  - match:
      router.status:
        query: 504
        type: phrase
  - match:
      applicationName:
        query: ingress-nginx
        type: phrase

Connection closed without response

On the Google Application Platform the error is signaled by HTTP status 502 in the application router logs.

Criteria to look for:

index: gap-app-name (replace with the proper index)
applicationName: ingress-nginx
router.status: 502

Client request interrupted

On the Google Application Platform the error is signaled by HTTP status 499 in the application router logs.

Criteria to look for:

index: gap-app-name (replace with the proper index)
applicationName: ingress-nginx
router.status: 499
router.bytes_sent: 0