Key Based Deduplication

Define dedup keys using customizable templates for configured alert sources. Auto-group similar incidents for efficient incident management and grouping of duplicates.

Key Based Deduplication is an efficient way to avoid duplicate entries when processing incoming Events alongside existing Incidents. It works by generating a Deduplication Key using a user-defined template specific to events from an Alert Source. This key helps identify and group duplicates.

Note: The introduction of this feature does not affect any existing Deduplication Rules you may have set up for your Services. Your current rules will remain unchanged and continue to function as expected, providing you with the same level of control and efficiency in managing your alerts.

If you switch to Key-Based Deduplication by setting it to active, Deduplication Rules will become inactive.

Important: Automation rule CRUD operations have a 5-minute caching delay before changes take effect.

How it works

1. To use the Key Based Deduplication feature, the user must opt-in to a specific Service.

2. They can define a template to generate dedupe keys for each alert source within the Service. The user also specifies a duration (x) for the Deduplication Window.

3. For an incoming Event, the Deduplication Key is calculated based on the defined template.

4. This Key is then compared (using equality) against any previous Incidents within the Deduplication Window (last 5 minutes or the specified duration).

5. If an Incident with the same Deduplication Key is found, the current Event is deduplicated against that Incident.

6. However, if no matching Incident is found, a new Incident is created.

7. Once the Deduplication Duration (x) elapses, the system recalculates the Deduplication Key using the defined template. This process continues for ongoing Deduplication.

Image. Flow Diagram for Key Based Deduplication

Prerequisites

The User Role associated with the user in the Team must have required permissions to manage Services (ability to manage Key Based Deduplication).

Create Key Based Deduplication

To add Key Based Deduplication:

1. Navigate to Services -> Service Overview -> Select or search for your desired Service.

2. On the extreme right, expand the accordion -> In the Automation section, View All.

3. In the Key Based Deduplication section, Add Dedup Key.

4. Select an alert source to begin creating Deduplication Keys for your incoming Events.

5. On the right, you can view the payload of the latest alert for the chosen Alert Source.

Note: You can configure one Dedup Key per Alert Source.

6. The user needs to define a template to generate Deduplication Keys using the variables from the payload referenced on the right, for a particular alert source of a Service. For additional information on how to write templates, please refer to Go's standard library. A Deduplication Key is calculated for the incoming event based on the template defined by the user.

Here are some illustrative examples that demonstrate how to define a template for generating Deduplication keys.

7. Define the Deduplication Time, in min(s) or hour(s).

Note: The maximum time limit is 48 hours.

8. Click Save.

Delete Key Based Deduplication

To delete a Key Based Deduplication config,

1. Click on the Key Based Deduplication Rule for a selected Service.

2. On the right-hand side, click More Options -> Delete Key

3. In the Configuration page, click Delete. A confirmation modal will appear.

4. Click Delete anyway to confirm.

Note: Kindly note that the Deduplication will cease once the Key is deleted.