Key Based Deduplication

Define dedup keys using customizable templates for configured alert sources. Auto-group similar incidents for efficient incident management and grouping of duplicates.

Key Based Deduplication is an efficient way to avoid duplicate entries when processing incoming Events alongside existing Incidents. It works by generating a Deduplication Key using a user-defined template specific to events from an Alert Source. This key helps identify and group duplicates.

Note: This feature is available across all pricing plans. For all older accounts (created before Aug 21, 2023) and Enterprise accounts:

The introduction of this feature does not affect any existing Deduplication Rules you may have set up for your Services. Your current rules will remain unchanged and continue to function as expected, providing you with the same level of control and efficiency in managing your alerts.

If you switch to Key-Based Deduplication by setting it to active, Deduplication Rules will become inactive.

Important: Automation rule CRUD operations have a 5-minute caching delay before changes take effect.

How it works

1. To use the Key Based Deduplication feature, the user must opt-in to a specific Service.

2. They can define a template to generate dedupe keys for each alert source within the Service. The user also specifies a duration (x) for the Deduplication Window.

3. For an incoming Event, the Deduplication Key is calculated based on the defined template.

4. This Key is then compared (using equality) against any previous Incidents within the Deduplication Window (last 5 minutes or the specified duration).

5. If an Incident with the same Deduplication Key is found, the current Event is deduplicated against that Incident.

6. However, if no matching Incident is found, a new Incident is created.

7. Once the Deduplication Duration (x) elapses, the system recalculates the Deduplication Key using the defined template. This process continues for ongoing Deduplication.

Image. Flow Diagram for Key Based Deduplication

Prerequisites

The User Role associated with the user in the Team must have required permissions to manage Services (ability to manage Key Based Deduplication).

Create Key Based Deduplication

To add Key Based Deduplication:

1. Navigate to Services -> Service Overview -> Select or search for your desired Service.

2. On the extreme right, expand the accordion -> In the Automation section, View All.

3. In the Key Based Deduplication section, Add Dedup Key.

4. Select an alert source to begin creating Deduplication Keys for your incoming Events.

5. On the right, you can view the payload of the latest alert for the chosen Alert Source.

Note: You can configure one Dedup Key per Alert Source.

6. The user needs to define a template to generate Deduplication Keys using the variables from the payload referenced on the right, for a particular alert source of a Service. For additional information on how to write templates, please refer to Go's standard library. A Deduplication Key is calculated for the incoming event based on the template defined by the user.

Here are some illustrative examples that demonstrate how to define a template for generating Deduplication keys.

7. Define the Deduplication Time, in min(s) or hour(s).

Regular Expression-Based Extraction in Go Template

Our system supports regular expression-based extraction using regex rules. It allows for multiple name captures but only retains the first match for a specific named group. Additionally, when the passed expression is not valid, the function returns empty match results.

{{- with $matches := ("(?m)^Container: (?P<container>.*)|Alertname: (?P<alertname>.*)|Summary: (?P<summary>.*)$" | reExtract .description) -}}
    {{$matches.container}}-{{$matches.alertname}}
{{- end -}}

This code snippet is a template written in Go's text templating language, used to parse alert descriptions. It extracts specific details like container name, alert name, and summary from the descriptions using regular expressions and presents them in a concise format.

Here's a breakdown:

1. Regular Expression ((?m)^Container: (?P<container>.*)|Alertname: (?P<alertname>.*)|Summary: (?P<summary>.*)$):

   • (?m) enables multiline matching.

   • ^Container: (?P<container>.*) captures everything after "Container: " into a named group "container".

   • |Alertname: (?P<alertname>.*) captures everything after "Alertname: " into a named group "alertname".

   • |Summary: (?P<summary>.*)$ captures everything after "Summary: " into a named group "summary" and ensures it matches the end of the line ($).

2. Template Processing ({{$matches := ...}}):

   • The code defines a variable $matches using the regular expression to extract details from the description string .description.

3. Output Formatting ({{$matches.container}}-{{$matches.alertname}})

   • The template accesses the captured container name ($matches.container) and alert name ($matches.alertname) from the $matches variable.

   • It combines them with a hyphen (-) for a one-line description format.

Note: The maximum time limit is 48 hours.

8. Click Save.

Delete Key Based Deduplication

To delete a Key Based Deduplication config,

1. Click on the Key Based Deduplication Rule for a selected Service.

2. On the right-hand side, click More Options -> Delete Key

3. In the Configuration page, click Delete. A confirmation modal will appear.

4. Click Delete anyway to confirm.

Note: Kindly note that the Deduplication will cease once the Key is deleted.

Have any questions? Ask the community.