API Filter Prometheus Examples

API Filter Prometheus Examples

Overview

This documentation provides step-by-step guidance on integrating Prometheus with Callgoose SQIBS automation platform using API filters. The example demonstrates how to configure custom filters for precise incident creation based on Prometheus alerts. For instance, you want incidents created only when any alert or event is from specific devices or systems.

Watch our quick video on API Filter Prometheus Examples for a detailed walkthrough

Steps to Integrate Prometheus

1. Navigate to API Integration

• Go to Services -> API Integration -> Add or Update API Integration.

1. Select API Integration

• Choose Prometheus from the Integration Template.
• The template automatically populates the required JSON values for Prometheus.

1. Default Behavior

• When Prometheus Alert Manager sends alerts to Callgoose SQIBS, incidents are created based on the alerts. By default, the following API filter values are added:

• Trigger:
• Urgency: High
• Payload JSON Key: status
• Key Value: firing
• Incident Mapping: groupKey
• Resolve:
• Incident Mapping: groupKey
• Payload JSON Key: status
• Key Value: resolved

1. Customizing Alerts for Specific Incidents

• To create incidents only for specific alerts or events (e.g., For instance, let’s say you want incidents created only when a specific website goes down. In your case, this could be any kind of alert or event from specific devices or a systems.), you can customize the API filters.

Viewing Prometheus Payload

Go to Service -> Add or Update API Integration

Right side bottom corner, you can see JSON Key Creator (purple circle icon).

1. View API Logs

• When Prometheus sends JSON data, it is logged in the API logs.
• Enable the debug option while creating the API token to access detailed logs.

1. Using JSON Key Creator

• Go to Services -> Add or Update API Integration.
• Use the JSON Key Creator (purple circle icon) to parse the payload.
• Example Payload:

{
  "receiver": "callgoose-sqibs",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "WebsiteDown",
        "instance": "https://incident-auto-remediation-sqibs.callgoose.com",
        "job": "blackbox",
        "severity": "critical"
      },
      "annotations": {
        "description": "The website https://incident-auto-remediation-sqibs.callgoose.com has been down for more than 1 minute.",
        "summary": "Website https://incident-auto-remediation-sqibs.callgoose.com is down"
      },
      "startsAt": "2024-09-30T10:51:27.939Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://f4d47a651869:9090/graph?g0.expr=probe_success+%3D%3D+0&g0.tab=1",
      "fingerprint": "6938de4b63636aad"
    }
  ],
  "groupLabels": {
    "alertname": "WebsiteDown"
  },
  "commonLabels": {
    "alertname": "WebsiteDown",
    "instance": "https://incident-auto-remediation-sqibs.callgoose.com",
    "job": "blackbox",
    "severity": "critical"
  },
  "commonAnnotations": {
    "description": "The website https://incident-auto-remediation-sqibs.callgoose.com has been down for more than 1 minute.",
    "summary": "Website https://incident-auto-remediation-sqibs.callgoose.com is down"
  },
  "externalURL": "http://526ab6abfd40:9093",
  "version": "4",
  "groupKey": "{}:{alertname=\"WebsiteDown\"}",
  "truncatedAlerts": 0
}

• Paste the payload into the JSON Key Creator to extract keys and values.
• Example Output:

 KEY                                                        VALUE
 status                                                     firing

alerts.[0].annotations.summary                              website https://incident-auto-remediation-runner-sqibs.callgoose.com:8443 is down

Adding Custom Filters

1. Customize Incident Creation

• Example:
• Add a second filter to create incidents only for specific alerts.
• Filter 1B:

Payload JSON Key: alerts.[0].annotations.summary

Key Value Contains: website https://incident-auto-remediation-sqibs.callgoose.com is down

Save the API filter.

Result: The incident is generated only if both filters match.

• Filter 1A:

Trigger: status contains firing

• Filter 1B:

alerts.[0].annotations.summary contains website https://incident-auto-remediation-sqibs.callgoose.com is down

1. Modify Incident Details

Example:

• Change the incident title:
• From: alerts.[0].annotations.summary
• Add a description:
• From: Entire payload data (default behavior).

1. Save the Customization

• Click the circle + icon to add additional filters and content.

Advanced Options

• Callgoose SQIBS provides advanced filtering options to customize incident creation based on specific alerts.
• Modify filters to control which alerts trigger incidents, ensuring only critical issues are escalated.

Conclusion

By following this documentation, you can efficiently integrate Prometheus with Callgoose SQIBS and customize API filters to streamline incident management. This setup allows fine-grained control over incident creation, tailored to your specific requirements.