LogoLogo
API DocsDeveloper PortalSystem StatusTry for Free
  • Quickstart Guide
    • Introduction
    • Get started as an Account Owner
    • Get started as a User
    • Glossary
    • FAQs
  • Manage Users
    • Types of Users
    • Add and Delete Users
    • Import Users
    • User Permissions - Access Controls
    • Manage Your Profile
    • Notification Rules
    • On-Call Reminder Rules
    • Change Account Owner
  • Manage Teams
    • Understanding Teams
    • Role Based Access Control
    • Owner Based Access Control
    • Create and Delete Teams
    • Add and Remove Team Members
    • Squads
    • Stakeholder Groups
  • Services
    • Adding a Service
    • Service Overview
    • Service Graph
    • Maintenance Mode
    • Alert Deduplication Rules
      • Alert Deduplication Rules
      • Incident Status Based Deduplication
      • Service Dependency Based Deduplication
      • Key Based Deduplication
    • Event Tagging
    • Alert Routing
    • Alert Suppression
    • Custom Content Templates
    • Intelligent Alert Grouping (IAG)
    • Auto Pause Transient Alerts (APTA)
    • Delayed Notifications
  • Schedules
    • Schedules (New)
      • Adding a Schedule
      • Schedules Overview
      • Who is On-Call?
      • My On-Call Shifts
      • Overrides
      • Videos: How to set up common use cases?
  • Escalation Policies
    • Create and Manage Escalation Policy
    • Round Robin & Advanced Escalations
    • Reassign an Incident
  • Notifications
    • Understanding Incident Notifications
  • Dashboards
    • Incident Management Dashboard
    • Dashboard Metrics
    • Take Bulk Actions
    • Squadcast Search
  • Incident List
    • Incident List View
    • Incident Priorities
    • Filter Incidents
    • Save Filter View
    • Merge Incidents
    • Snooze Incidents
  • Incidents Page
    • Incidents Details
    • Incident Activity Timeline
    • Communication Channels
    • Create Incident Manually
    • Incident Notes
    • Incident Watchers
    • Past Incidents
    • Additional Responders
    • Incident Summaries
    • Incident Suggestions
  • Runbooks
    • Runbooks
  • Postmortems
    • Postmortem Templates
    • Create Postmortems
    • Accessing Postmortem
  • Status Page
    • Status Page
    • Status Page Overview
    • Components and Groups
    • Issues
    • Subscribers
    • Maintenance
  • SLO Tracker
    • SLO Basics
    • Configure and Monitor your SLOs
  • Webforms
    • Webforms
  • Global Event Rulesets
    • Global Event Rulesets
  • Workflows
    • Workflows
    • Workflows Overview
    • Actions
  • Live Call Routing
    • Live Call Routing
  • Analytics
    • Analytics (New)
    • Organization Level Analytics
    • On Call Hours Per User
    • Weekly Reports
  • Integrations
    • Incident Webhook (Incident Webhook/API)
    • Outgoing Webhooks
    • ServiceNow Extension
    • Extensions
      • Jira Cloud Integration
      • Jira DC (Data Center)
      • CircleCI
      • Google Chat
      • Freshdesk
      • Freshservice
      • Asana
      • ClickUp
      • Trello
      • Zendesk
      • Hubspot
    • Alert Source Integrations (Native)
      • Admin Labs
      • Airbrake
      • Amazon EventBridge
      • Amazon GuardDuty
      • Amazon Opensearch
      • APImetrics
      • AppDynamics
      • AppSignal
      • Auvik
      • AWS CloudTrail Logs
      • AWS CloudTrail via CloudWatch
      • Amazon Cloudwatch (AWS) Integration
      • AWS CloudWatch Event Rules
      • AWS Elastic Beanstalk via CloudWatch
      • Amazon RDS (AWS)
      • Amazon SNS (AWS)
      • Azure Monitor
      • Better Uptime
      • Bitbucket
      • Bitrix 24
      • Blue Matador
      • Bugsnag
      • Buildkite
      • Checkly
      • Checkmk
      • CircleCI Integration
      • Cisco DNAC
      • Cisco Meraki
      • ClickUp Integration
      • CloudAMQP
      • Cloudflare
      • Conviva
      • CopperEgg
      • Coralogix
      • Cronitor
      • Crowdstrike Falcon
      • Datadog
      • Databricks
      • Dead Man's Snitch
      • Domotz
      • Dotcom Monitor
      • Dynatrace
      • ElastAlert
      • Elastic
      • Elecard Boro
      • Email Integration
      • Endtest
      • Errorception
      • Freshdesk Integration
      • Freshping
      • Freshservice
      • Ghost Inspector
      • GitHub Integration
      • GitLab
      • Grafana 8
      • Grafana
      • Graylog v4
      • Graylog
      • HaloPSA
      • Healthchecks
      • Heroku
      • HetrixTools
      • Honeybadger
      • Honeycomb
      • Humio
      • Hund
      • Hydrozen
      • Hyperping
      • Icinga2
      • InsightOps (LogEntries)
      • Instana
      • Intercom
      • Jenkins Integration
      • Jira Cloud Alert Source
      • Jira Server Alert Source
      • Kapacitor
      • Kentik
      • Komodor
      • Kibana
      • LibreNMS
      • Linear
      • Loggly
      • Logstash
      • Logz.io
      • ManageEngine Application Manager
      • ManageEngine Opmanager
      • Mezmo (formerly LogDNA)
      • MongoDB Atlas / Cloud Manager
      • Nagios
      • New Relic
      • Nixstats
      • NodePing
      • Observium
      • Oh Dear
      • Oracle Cloud Infrastructure
      • OSNexus QuantaStor
      • OverOps
      • Papertrail
      • Pingdom
      • Plesk 360
      • Postman
      • Postmark
      • Powercode
      • Progress WhatsUp Gold
      • Prometheus
      • PRTG Network Monitor
      • Rapid7 InsightIDR
      • RapidSpike
      • Redash
      • Redgate SQL Monitor
      • Rollbar
      • Rundeck
      • Runscope
      • Salesforce Cloud
      • Scout APM
      • Sematext
      • Sensu Go
      • Sensu
      • Sentry.io
      • Server Density
      • ServerGuard24
      • ServiceNow Integration
      • Shortcut (Clubhouse)
      • SignalFx
      • SigNoz
      • Site24x7
      • Slack
      • SolarWinds AppOptics
      • SolarWinds Observability SaaS (SWO)
      • SolarWinds Orion
      • Sonar
      • Splunk
      • Sqreen
      • Stackdriver
      • Stackify Retrace
      • StatHat
      • StatusCake
      • ServiceDesk Plus OD
      • Sumo Logic
      • Sysdig Monitor
      • Threat Stack
      • Trello
      • Twilio
      • Uptime
      • Uptime Robot
      • Uptrends
      • Wavefront
      • Zabbix 5.0
      • Zabbix 6.2
      • Zabbix
      • Zendesk Integration
      • Zoho Desk
      • Zoho Desk via Zoho Flow
      • LogicMonitor
  • ChatOps
    • Google Chat
    • Microsoft Teams
    • Slack for Incident Management
      • Using the Integration
  • Single Sign-On (SSO)
    • AWS SSO
    • Azure Active Directory SSO
    • Google SSO
    • Microsoft ADFS SSO
    • Okta SSO Integration
    • SAML 2.0 based SSO
  • Mobile App
    • Using the Mobile App
  • Terraform & API Documentation
    • Terraform Provider
    • Public API - Refresh Token
    • API Documentation
    • Getting Started with Squadcast GraphQL
      • Schedules
        • Create Schedule
        • Update Schedule
        • Delete Schedule
        • Pause Schedule
        • Get Schedules
        • Get Schedule by ID
        • Resume Schedule
        • Clone Schedule
        • Get Gaps
      • Rotations
        • Create Rotation
        • Update Rotation
        • Delete Rotation
        • Get Rotation by ID
        • Get Rotation Events by ID
      • Overrides
        • Create Override
        • Update Override
        • Delete Override
        • Get Override by ID
      • Calendar URLs
      • Who is On-Call
    • Developer Portal
    • Incident Rate Limiting
  • Managing your Squadcast Account
    • Audit Logs
    • Organizations
    • Billing FAQs
    • Deactivate your Squadcast Account
    • Delete your Squadcast Account
Powered by GitBook
On this page
  • Prerequisites
  • Create a Deduplication Rule
  • A. UI-based Rule Builder (Beginner-friendly)
  • B. Raw String Method
  • Viewing Deduplicated Incidents
  • FAQs

Was this helpful?

  1. Services
  2. Alert Deduplication Rules

Alert Deduplication Rules

Reduce alert noise by grouping similar alerts together

PreviousAlert Deduplication RulesNextIncident Status Based Deduplication

Last updated 2 months ago

Was this helpful?

Alert Deduplication can help you reduce alert noise by organizing and grouping alerts. This also provides easy access to similar alerts when needed.

This can be achieved by defining Deduplication Rules for each Service in Squadcast. When these rules evaluate true for an incoming incident, alerts will get deduplicated.

Prerequisites

  • The User Role associated with the user in the Team must have required permissions to manage Services (ability to manage Deduplication Rules).

  • Integrate with an Alert Source and ensure that the Alert Source has started sending alerts to Squadcast before setting up Deduplication Rules.

Note: The Key-Based Deduplication (Dedupe Keys) feature is available across all subscription plans.

Regarding Deduplication Rules Feature:

For Existing Accounts:

  • No changes will be applied, and the current settings will remain in effect.

For New Sign-ups (Post Deduplication Keys Release - Aug 21, 2023):

  • Deduplication Rules will only be accessible for enterprise accounts. Other subscription plans will not have access to Deduplication Rules.

This clarification aims to ensure a better understanding of the deduplication framework and its application across different account scenarios.

Important:

  • Deduplication Rules work only on incidents in either the Triggered or Acknowledged states.

  • Automation rule CRUD operations have a 5-minute caching delay before changes take effect.

Create a Deduplication Rule

To create a Deduplication Rule:

  1. Navigate to Services -> Service Overview -> select or search for your desired service.

  2. In the extreme right, expand the accordion -> In Automation section, View All

  3. In the Automation Rules section, Add Deduplication Rules

  4. Select an Alert Source from the drop down -> Add New Rule

  5. Deduplication Rules can be added in two ways:

A. UI-based Rule Builder (Beginner-friendly)

  1. On the right, you can view the payload of the latest alert for the chosen Alert Source

  2. The drop-downs in the Rule Builder contain values from the payload on the right. You can use them to easily create your Deduplication Rules

You can create Deduplication Rules using the following conditions:

Operators
Condition

==

if the payload value is equal to the given value

!=

if the payload value is not equal to the given value

matches/contains

if the payload value matches (or contains) the given value

does not contain

if the payload value does not contain the given value

Note: All these operators are case-sensitive. If you want to make the rules case insensitive, then you have to do it with the regular expression method.

3. You can add more than 1 condition for a rule by selecting Add Condition (a logical AND is performed between all the conditions -> the entire Deduplication Rule will evaluate to True only if all the conditions evaluate to True)

Next, choose the Deduplication Time Window. You can deduplicate incidents for a maximum of 48 hours.

Time Window

During the set time window, incidents that occur are compared against all incidents that come in during that time period, and will then get deduplicated against the first incident that it matches with.

4. Click on Save

Note:

The search option under payload is not a free search, we have to search by JSON format, for example, type in payload.annotations to get annotations.

We also have an option for click-to search, wherein you can click on the keys in the payload to get their required values.

B. Raw String Method

Important

Once you opt for the Raw String method, you cannot revert to the UI-based Rule Builder method for that rule.

  1. On the right, you can view the payload of the latest alert for the chosen Alert Source

  2. Click on Edit to enable the Raw String method

3. Write your custom Deduplication Rule expression. Below are some examples to help you get started:

How to make rules case-insensitive?

  1. Click Edit -> Proceed

  2. You will see the rule in the regular expression. Now you need to add the command lc (lower_case) before the individual parameters

Here is an example:

re(payload["subscription"]["type"], "Subscription")

The rule says if the payload["subscription"]["type"]contains the string "Subscription" in it then do some actions.

If you want to make the above rule case insensitive, you have to add the command lc before the individual parameters.

The case insensitive rule would look like this,

re(lc(payload["subscription"]["type"]), lc("Subscription")) or

re(lc(payload["subscription"]["type"]), "subscription")

Supported Rules

The rule engine supports expressions with parameters, arithmetic, logical, and string operations.

Basic Expressions

10>0,1+2,100/310 > 0, 1+2, 100/310>0,1+2,100/3

Parameterized Expressions

past.metric==current.metricpast.metric == current.metricpast.metric==current.metric

The available parameters are past, current and event_count + past: This parameter contains the JSON payload of the previous incident which the current event is compared with + current: This parameter contains the JSON payload of the incoming event which will be compared with the past incidents’ JSON payload + event_count: This denotes the number of deduplicated events for a given incident

Use-case for event_count:

This can be used in scenarios where you do not want to deduplicate more than n number of alerts to a particular incident.

Regular Expressions

This can be used to check if a particular JSON payload field matches a regular expression.

re(payload.metric,"disk.∗")re(payload.metric, "disk.*")re(payload.metric,"disk.∗")

Parse JSON content within the payload using jsonPath

General Format

jsonPath(<the JSON string that should be parsed for JSON content>, <"the parameter that needs to be picked from the parsed JSON object">)

Example

Below is an example payload:

{
	"payload": {
		"Type": "Notification",
		"MessageId": "5966c484-5b37-58df",
		"TopicArn": "arn:aws:sns:us-east-1:51:Test",
		"Message": "{\"AlarmName\":\"Squadcast Testing - Ignore\",\"AlarmDescription\":\"Created from EC2 Console\"}"
	}
}
jsonPath(payload.Message, "AlarmName");

This will pick out the value AlarmName from the Message object in the payload, based on which, you can de-duplicate incidents.

Multiple Alert Sources

We can see alert payloads of past events from different alert sources for the Service by selecting the respective alert source from the dropdown in the right-half side.

Since the payload format is fixed for a given alert source, it is usually preferable to have Deduplication Rules on a per-alert source basis. This can be done by making use of the source field which lets you know the alert source that triggered the incoming event. For example, if you want to have a Deduplication Rule for a Service, only for alerts coming for grafana alert source, then the corresponding rule would look something like: source == 'grafana' && (<your_deduplication_rule>)

Example

Below is an example payload for demonstration:

{
	"event_count": 5,
	"past": {
		"issue_description": "bug - 2",
		"issue_id": "10029",
		"issue_key": "HYD-30",
		"issue_labels": [],
		"issue_link": "http://13.233.254.18:8080/browse/HYD/issues/HYD-30",
		"issue_priority": "Medium",
		"issue_summary": "bug - 2",
		"issue_type": "Bug",
		"project_id": "10000",
		"project_key": "HYD",
		"project_name": "hydra"
	},
	"source": "grafana"
}

To deduplicate any incoming alert when:

  • The metric matches the regular expression ^disk.*

  • The past incident metric and the current event metric are the same

  • The past incident host and the current event host are the same

  • The current disk usage value is less than 60%

  • The context value tag is same

Deduplication Rule:

past.metric == current.metric &&
	re(current.metric, "^disk.*") &&
	past.host == current.host &&
	current.value < 60 &&
	jsonPath(past.tags, "context.value") ==
		jsonPath(current.tags, "context.value");

Viewing Deduplicated Incidents

The number indicates how many alerts were deduplicated against this incident.

  • Number of deduplicated events

  • Time when they reached Squadcast

  • Message and Payload of the event

Clicking on any of the deduplicated events will display will all the information that is sent for that alert from the monitoring tool, including the Deduplication Reason (which Deduplication Rule got executed).

FAQs

1. Will I get notified of the duplicate alerts that come in for an incident in a Triggered state?

No, nobody is notified of the duplicate alerts that come in for an incident in the Triggered or Acknowledged state.

2. Will I get notified of the duplicate alerts that come in for an incident in the Resolved state?

Yes, Incident Deduplication will not take place for an incident when it is in the Resolved state. Squadcast triggers a fresh incident for such an alert and notifies the right people.

3. Why can’t I deduplicate incidents for more than 48 hours?

Most organizations across the world follow the best practice of resolving critical incidents within 48 hours of their creation. This is also the reason why Squadcast allows you to deduplicate incidents for a maximum of 48 hours.

4. Can I create OR rules?

Yes, you can. The evaluation between different Deduplication Rules is OR.

5. What kind of regex can be used to write custom rules?

6. Do the Deduplication Rules have priority?

Yes, you can specify Execution Rule Priority for the rules defined by moving them Up or Down the list of rules.

7. While adding a Deduplication Rule, is the search string in the rule case sensitive?

Yes, that is correct. For example, if your search string is “ALERT” and your payload does not contain “ALERT” but contains “Alert”, this will not be matched. Your search string should be “Alert”.

8. I have configured multiple rules for a particular Service. Can I search through the configured rules to find the rule I am looking for?

Yes, that is doable. You will notice a Search option on the left-top of the rules modal. You can type in a word you recall from the rule description or the rule itself. Any matching results will yield the narrowed-down set of rules.

From the page, you can view which incidents have deduplicated events when there is a +<number> next to the Incident ID like in the screenshot below.

Clicking on such an incident will take you to its page where, by clicking on Deduped Events, you will be able to see the following:

The rule engine supports expressions with parameters, arithmetic, logical, and string operations. You can also check out to get an idea of all the expression types accepted in Squadcast. Please do your regex against Golang flavor as shown in the screenshot below and then, set them up in Squadcast:

Have any questions? .

Incidents
Incident Details
this
here
Ask the community
Alert Deduplication based on time in Squadcast
Add alert deduplication based on time in Squadcast
deduplication rule in Squadcast using raw string method
Viewing deduplicated incidents in Squadcast
deduplicated events in Squadcast
Search rules configured in Squadcast