Prometheus

Send events to Squadcast from Prometheus

Follow the steps below to configure a service so as to extract its related alert data from Prometheus.

Squadcast will then process this information to create incidents for this service as per your preferences.

Using Prometheus as an Alert Source

1. Navigate to Services -> Service Overview -> select or search for your Service. Expand the accordion -> In the Alert Sources section, click Add.

2. Select Prometheus. Copy the displayed Webhook URL to configure it within Prometheus. Finish by clicking Add Alert Source -> Done.

Important:

When an alert source turns Active, it’ll show up under Configured Alert Sources, you can either generate a test alert from the integration or wait for a real-time alert to be generated by the Alert Source. An Alert Source is active if there is a recorded incident via that Alert Source for the Service.

Create a Squadcast Webhook in Prometheus

Now open your Prometheus Alertmanager's Configuration file. (You need Prometheus Alertmanager to handle alerts generated by Prometheus. Click here for more information )

Add a new webhook receiver and enter the URL obtained earlier as the webhook url.

Now you can use this receiver in any of your alert routes (You can also make it the default receiver, as per your requirements).

Also please make sure that your alerts has the severity under labels, and annotations has the summary & description fields configured in the alert rules file as shown below in the example rules config.

serverFiles:
  alerts:
    groups:
    - name: gt.k8s.kafka.rules
      rules:
      - alert: GtK8sKafkaConsumerLag1Warning
        expr: kafka_consumergroup_lag > 500
        for: 120m
        labels:
          severity: warning
        annotations:
          description: "Kafka consumer { { $labels.consumergroup } } lags with { { $value } } messages in topic { { $labels.topic } }"
          summary: "Kafka affected in { { $labels.instance } }"

Important:

We use the summary field along with Severity & Alert group name as incident message and the description field along with generator url as incident description. Hence, severity (labels), summary & description (annotations) are mandatory for integration with Squadcast.

Note:

Any alert which gets resolved from Prometheus will automatically be resolved inside Squadcast as well, unless you have explicitly set send_resolved to false in the config file.