Key Based Deduplication

Define dedup keys using customizable templates for configured alert sources. Auto-group similar incidents for efficient incident management and grouping of duplicates.
Key-based Deduplication is an efficient way to avoid duplicate entries when processing incoming Events alongside existing Incidents. It works by generating a Deduplication Key using a user-defined template specific to events from an Alert Source. This key helps identify and group duplicates.

How it works

  1. 1.
    To use the key-based deduplication feature, the user must opt-in to a specific Service.
  2. 2.
    They can define a template to generate dedupe keys for each alert source within the Service. The user also specifies a duration (x) for the Deduplication Window.
  3. 3.
    For an incoming Event, the Deduplication Key is calculated based on the defined template.
  4. 4.
    This Key is then compared (using equality) against any previous Incidents within the Deduplication Window (last 5 minutes or the specified duration).
  5. 5.
    If an Incident with the same Deduplication Key is found, the current Event is deduplicated against that Incident.
  6. 6.
    However, if no matching incident is found, a new Incident is created.
  7. 7.
    Once the Deduplication Duration (x) elapses, the system recalculates the Deduplication Key using the defined template. This process continues for ongoing deduplication.
Flow Diagram for Key Based Deduplication
Image. Flow Diagram for Key Based Deduplication


The User Role associated with the user in the Team must have required permissions to manage Services (ability to manage Key Based Deduplication).

Configure Key Based Deduplication

To add key-based deduplication:
  1. 1.
    Navigate to Services -> Service Overview -> select or search for your desired Service.
  2. 2.
    On the extreme right, expand the accordion -> In the Automation section, View All.
  3. 3.
    In the Key Based Deduplication section, Add Dedup Key.
  4. 4.
    Select an alert source to begin creating Deduplication Keys for your incoming Events.
  5. 5.
    On the right, you can view the payload of the latest alert for the chosen Alert Source.
Note: You can configure one Dedup Key per Alert Source.
  1. 6.
    The user needs to define a template to generate De-duplication Keys using the variables from the payload referenced on the right, for a particular alert source of a Service. For additional information on how to write templates, please refer to Go's standard library. A De-duplication key is calculated for the incoming event based on the template defined by the user.
Here are some illustrative examples that demonstrate how to define a template for generating De-duplication keys.
  1. 7.
    Define the Deduplication Time, in min(s) or hour(s).
Note: The maximum time limit is 48 hours.
  1. 8.
    Click Save.

Delete Key Based Deduplication

To delete a key-based deduplication config:
  1. 1.
    Click on the Key Based Dedup Rule for a selected Service.
  2. 2.
    On the right-hand side, click More Options -> Delete
  3. 3.
    Click Delete again to confirm.
Note: Kindly note that the deduplication will cease once the key is deleted.