Alert Suppression
Fight alert fatigue with Alert Suppression - Learn how to optimize incident management by effectively suppressing non-actionable notifications. Take control today!
Last updated
Fight alert fatigue with Alert Suppression - Learn how to optimize incident management by effectively suppressing non-actionable notifications. Take control today!
Last updated
Alert Suppression can help you avoid alert fatigue by suppressing notifications for non-actionable alerts.
Squadcast will suppress the incidents that match any of the Suppression Rules you create for your Services. These incidents will go into the Suppressed state and you will not get any notifications for them.
These are useful in situations where you would like to view your all your informational alerts in Squadcast but do not want to get notified for them.
The User Role associated with the user in the Team must have required permissions to manage Services (ability to manage Suppression Rules).
Integrate with an Alert Source and ensure that the Alert Source has started sending alerts to Squadcast before setting up Suppression Rules.
Important: Automation rule CRUD operations have a 5-minute caching delay before changes take effect.
Navigate to Services -> Service Overview -> select or search for your desired service.
In the extreme right, expand the accordion -> In Automation section, View All.
In the Suppression Rules section, Add Suppression Rules.
Select an Alert Source from the drop down -> Add New Rule.
5. Suppression Rules can be added in two different ways:
1. On the right, you can view the payload of the latest alert for the chosen Alert Source
2. The drop-downs in the Rule Builder contain values from the payload on the right. You can use them to easily create your Suppression Rules. As you build the expression from these drop-downs, you can also see the corresponding raw string being auto-populated for the same under String Expression.
You can create Suppression Rules using the following conditions:
==
if the payload value is equal to the given value
!=
if the payload value is not equal to the given value
matches/contains
if the payload value matches (or contains) the given value
does not contain
if the payload value does not contain the given value
Note: All these operators are case-sensitive. If you want to make the rules case insensitive, then you have to do it with the regular expression method.
3. You can add more than 1 condition for a rule by selecting Add Condition (a logical AND is performed between all the conditions -> the entire Suppression Rule will evaluate to True only if all the conditions evaluate to True)
Note:
The drop-down blocks only support the logical AND operator between 2 expressions. If you want to have a logical OR operation between 2 expressions, then you would have to create a new Suppression Rule.
Comparison Operators within Suppression Rules
You can also leverage comparison operators such as ==, <, <=, >, >= within your rules using the drop-down blocks, when the parameter you are evaluating against, is a numerical value from the payload to reduce alert noise.
4. You can suppress incidents based on time as well. To do so, check to Suppress by time. Add details for your suppression time slots like Timezone, Duration and Repetition.
Under Duration, you can specify Start and End Dates and choose Start and End Time as well or simply run it for the entire day.
You can add Repetition for your slot, to do so, you can choose from the drop-down list, while specifying the end for this repetition, as a particular date/time or never.
Note: You can add multiple suppression time slots for a single Suppression Rule.
Note: Users can select a timezone as per which the time slot needs to be active.
Note: The search option under payload is not a free search, we have to search by JSON format, for example, type in payload.annotations to get annotations.
We also have an option for click-to search, wherein you can click on the keys in the payload to get their required values.
Important
Once you opt for the Raw String method, you cannot revert to the UI-based Rule Builder method.
(a) On the right, you can view the payload of the latest alert for the chosen Alert Source
(b) Click on Edit to enable the Raw String method
(c) Write your custom Suppression Rule expression
How to make rules case-insensitive?
Click Edit -> Proceed
You will see the rule in the regular expression. Now you need to add the command lc (lower_case) before the individual parameters
Here is an example:
re(payload["subscription"]["type"], "Subscription")
The rule says if the payload["subscription"]["type"]contains the string "Subscription" in it then do some actions.
If you want to make the above rule case insensitive, you have to add the command lc before the individual parameters.
The case insensitive rule would look like this,
re(lc(payload["subscription"]["type"]), lc("Subscription")) or
re(lc(payload["subscription"]["type"]), "subscription")
(d) You can suppress incidents based on time as well. To do so, check to Suppress by time. Add details for your suppression time slots like Timezone, Duration and Repetition.
Under Duration, you can specify Start and End Dates and choose Start and End Time as well or simply run it for the entire day.
You can add Repetition for your slot, to do so, you can choose from the drop-down list, while specifying the end for this repetition, as a particular date/time or never.
Note: You can add multiple suppression time slots for a single Suppression Rule.
Note: Users can select a timezone as per which the time slot needs to be active.
The rule engine supports expressions with parameters, arithmetic, logical, and string operations. You can also check out this link to get an idea of all the expression types accepted in Squadcast.
10 > 0, 1+2, 100/3
payload.metric == "disk" The available parameters are payload: This parameter contains the JSON payload of an incident which will be the same as the JSON payload format for the future events for a particular alert source payload: This parameter contains the JSON payload of an incident which will be the same as the JSON payload format for the future events for a particular alert source payload: This parameter contains the JSON payload of an incident which will be the same as the JSON payload format for the future events for a particular alert source incident_details: This contains the content of the message and description of the incoming event source: This denotes the associated alert source for the current/incoming event
re(payload.metric, "disk.*")
jsonPath to add a tagjsonPath(<the JSON string that should be parsed for JSON content>, <"the parameter that needs to be picked from the parsed JSON object">)Below is an example payload:
This will pick out the value AlarmName from the Message object in the payload based on which, you can suppress the incident.
Multiple Alert Sources
We can see alert payloads of past events from different alert sources for the service by selecting the respective alert source from the dropdown on the right-half side.
Since the payload format is fixed for a given alert source, it is usually preferable to have suppression rules on a per-alert source basis. This can be done by making use of the source field which lets you know the alert source that triggered the incoming event.
For example, if you want to have a suppression rule for a service, only for alerts coming from grafana alert source, then the corresponding rule would look something like this:
Below is an example payload for demonstration:
To suppress any incoming alert when:
The alert message contains: [Bug]
The alert source is grafana
Suppression Rule:
To discard incoming alerts and stop them from being triggered as incidents in Squadcast, use the discard() function in conjunction with Suppression Rules.
Suppression Rule:
Suppression Rule with discard():
Avoid hitting Rate Limits
The discard() function can be used to avoid hitting the Incident Rate Limits as Suppressed events that are discarded don’t get counted against the allowed rate limits.
You can view suppressed incidents on the Incidents page by clicking on All Incidents and choosing Suppressed as highlighted in the screenshot below.
Note
Suppressed and Resolved are the final states for incidents in Squadcast. You will not be able to take any action on incidents that are in these states.
Incident information will be available on the Squadcast platform even if they are suppressed.
1. What kind of regex can be used to write custom rules?
The rule engine supports expressions with parameters, arithmetic, logical, and string operations. You can also check this out to get an idea of all the expression types accepted in Squadcast. Please do your regex here against Golang flavour as shown in the screenshot below and then, set them up in Squadcast:
2. Can I create OR rules?
Yes, you can. The evaluation between different Suppression Rules is OR. Add multiple Suppression Rules to enable OR evaluation.
3. While adding a Suppression Rule, is the search string in the rule case sensitive?
Yes, that is correct. For example, if your search string is “ALERT” and your payload does not contain “ALERT” but contains “Alert”, this will not be matched. Your search string should be “Alert”.
4. How do I know if an incident gets suppressed due to a Suppression Rule?
In the Incident’s Activity Timeline, the reason for suppression is displayed.
5. I have configured multiple rules for a particular Service. Can I search through the configured rules to find the rule I am looking for?
Yes, that is doable. You will notice a Search option on the left top of the rules modal. You can type in a word you recall from the rule description or the rule itself. Any matching results will yield a narrowed-down set of rules.
