Integrations

SigNoz

Overview

This document provides a detailed guide to integrating SigNoz with Callgoose SQIBS for real-time Incident Management, Incident Auto Remediation, Event-Driven Automation, and other automation purposes. The integration enables automatic creation, updating, and resolution of incidents in Callgoose SQIBS based on alerts triggered in SigNoz. The guide includes steps for setting up triggers in SigNoz, configuring webhook notifications, creating API filters in Callgoose SQIBS, and troubleshooting.

1. Obtain API Token and Endpoint Details

To integrate with Callgoose SQIBS, you first need to obtain an API token and find the API endpoint details.

Generate an API Token:
Follow the guide on How to Create API Token in Callgoose SQIBS.
Find the API Endpoint:
Refer to the Callgoose SQIBS API Endpoint Documentation to get the endpoint details where the JSON payloads from SigNoz will be sent.

2. Debugging and Troubleshooting

You can enable debugging in the API tokens used with SigNoz notifications for troubleshooting purposes.

Enable Debugging:
You can update the debug value when adding or updating an API token.
When API tracking is enabled, logs are stored in the API log section for your review. The debugging option will automatically disable after 48 hours.
When API tracking is turned off, no logs are saved in the API log.
Using API Log for Troubleshooting:
The API log provides detailed information on all API calls made to Callgoose SQIBS.
You can check the JSON values in each API log entry for troubleshooting purposes.
Use the information in the API log to create or refine API filters to ensure incidents are created correctly based on the API payloads received.
Callgoose SQIBS creates incidents according to your API filter configuration, giving you full control over how alerts from different services trigger incidents and alerts for your support team or automation processes.

3. Configuring SigNoz to Send JSON Payloads

To configure SigNoz to generate the JSON payloads similar to the examples provided, follow the steps outlined below. These steps will guide you through setting up the necessary triggers and webhook notifications within SigNoz to ensure that the JSON payloads match those expected by Callgoose SQIBS.

3.1 Setting Up Webhook in SigNoz

Log into SigNoz
Open SigNoz and log in using your credentials.
Navigate to Getting Started.
Follow the on-screen instructions to connect your application or infrastructure to SigNoz.
Configure Webhook Alert Channel
After connecting your application, go to Settings.
Click on Alert Channels.
Click on New Channel.
Provide a name for the channel.
Select Webhook as the channel type.
In the Webhook URL field, enter the Webhook URL endpoint provided by Callgoose SQIBS.
Refer to the Callgoose SQIBS API End Point Documentation for the final API endpoint. This may vary based on authentication and other URL parameters.
Click the Test button to test the connection.
Verify the test payload in the API Logs of Callgoose SQIBS.
Once the test is successful, click Save.

3.2 Setting Up Alert Rule in SigNoz

Create an Alert Rule
Navigate to Alerts in SigNoz.
Click on New Alert Rule.
Choose the type of alert you want to configure (e.g., Metric-based alert).
Follow the instructions provided on the page for configuration details.
Customize the alert settings according to your requirements.
Scroll down to the Notification Channel field.
Select the Webhook channel created earlier.
Test the notification by clicking Test Notification.
After configuring the alert metrics, click Create Rule.

3.3 Finalizing and Testing

Validate the Integration:
Trigger the condition manually if possible to verify that the correct JSON payload is sent to Callgoose SQIBS.
Resolve the alert to ensure the resolved state payload is also correctly sent and processed.

4. Configuring Callgoose SQIBS

4.1 Create API Filters in Callgoose SQIBS

To correctly map incidents from the SigNoz alerts, you need to create API filters based on the JSON payloads received.

4.1.1 Example JSON Payloads from SigNoz

Alert Triggered (status: "firing")

json

{
  "receiver": "Callgoose",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "Test",
        "device": "dm-0",
        "ruleId": "1",
        "ruleSource": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
        "severity": "critical"
      },
      "annotations": {
        "description": "This alert is fired when the defined metric (current value: 23.183333333333334) crosses the threshold (20)",
        "summary": "The rule threshold is set to 20, and the observed metric value is 23.183333333333334"
      },
      "startsAt": "2024-11-21T07:34:23.708306749Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
      "fingerprint": "3dbd493b1878ab2b"
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "Test",
        "device": "dm-2",
        "ruleId": "1",
        "ruleSource": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
        "severity": "critical"
      },
      "annotations": {
        "description": "This alert is fired when the defined metric (current value: 23.183333333333334) crosses the threshold (20)",
        "summary": "The rule threshold is set to 20, and the observed metric value is 23.183333333333334"
      },
      "startsAt": "2024-11-21T07:34:23.708306749Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
      "fingerprint": "9b9d7913f429b9a5"
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "Test",
        "device": "nvme0n1",
        "ruleId": "1",
        "ruleSource": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
        "severity": "critical"
      },
      "annotations": {
        "description": "This alert is fired when the defined metric (current value: 22.566666666666666) crosses the threshold (20)",
        "summary": "The rule threshold is set to 20, and the observed metric value is 22.566666666666666"
      },
      "startsAt": "2024-11-21T07:34:23.708306749Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
      "fingerprint": "2996a49f3d92b71e"
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "Test",
        "device": "nvme0n1p7",
        "ruleId": "1",
        "ruleSource": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
        "severity": "critical"
      },
      "annotations": {
        "description": "This alert is fired when the defined metric (current value: 22.566666666666666) crosses the threshold (20)",
        "summary": "The rule threshold is set to 20, and the observed metric value is 22.566666666666666"
      },
      "startsAt": "2024-11-21T07:34:23.708306749Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
      "fingerprint": "eb584b7671528e35"
    }
  ],
  "groupLabels": {
    "alertname": "Test"
  },
  "commonLabels": {
    "alertname": "Test",
    "severity": "critical"
  },
  "commonAnnotations": {},
  "externalURL": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
  "version": "4",
  "groupKey": "{}/{}:{alertname=\"Test\"}",
  "truncatedAlerts": 0
}

Alert Resolved (status: "resolved")

json

{
  "receiver": "Callgoose",
  "status": "resolved",
  "alerts": [
    {
      "status": "resolved",
      "labels": {
        "alertname": "Test",
        "device": "dm-0",
        "ruleId": "1",
        "ruleSource": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
        "severity": "critical"
      },
      "annotations": {
        "description": "This alert is fired when the defined metric (current value: 23.183333333333334) crosses the threshold (20)",
        "summary": "The rule threshold is set to 20, and the observed metric value is 23.183333333333334"
      },
      "startsAt": "2024-11-21T07:34:23.708306749Z",
      "endsAt": "2024-11-21T07:39:23.708306749Z",
      "generatorURL": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
      "fingerprint": "3dbd493b1878ab2b"
    },
    {
      "status": "resolved",
      "labels": {
        "alertname": "Test",
        "device": "dm-2",
        "ruleId": "1",
        "ruleSource": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
        "severity": "critical"
      },
      "annotations": {
        "description": "This alert is fired when the defined metric (current value: 23.183333333333334) crosses the threshold (20)",
        "summary": "The rule threshold is set to 20, and the observed metric value is 23.183333333333334"
      },
      "startsAt": "2024-11-21T07:34:23.708306749Z",
      "endsAt": "2024-11-21T07:39:23.708306749Z",
      "generatorURL": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
      "fingerprint": "9b9d7913f429b9a5"
    },
    {
      "status": "resolved",
      "labels": {
        "alertname": "Test",
        "device": "nvme0n1",
        "ruleId": "1",
        "ruleSource": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
        "severity": "critical"
      },
      "annotations": {
        "description": "This alert is fired when the defined metric (current value: 22.566666666666666) crosses the threshold (20)",
        "summary": "The rule threshold is set to 20, and the observed metric value is 22.566666666666666"
      },
      "startsAt": "2024-11-21T07:34:23.708306749Z",
      "endsAt": "2024-11-21T07:39:23.708306749Z",
      "generatorURL": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
      "fingerprint": "2996a49f3d92b71e"
    },
    {
      "status": "resolved",
      "labels": {
        "alertname": "Test",
        "device": "nvme0n1p7",
        "ruleId": "1",
        "ruleSource": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
        "severity": "critical"
      },
      "annotations": {
        "description": "This alert is fired when the defined metric (current value: 22.566666666666666) crosses the threshold (20)",
        "summary": "The rule threshold is set to 20, and the observed metric value is 22.566666666666666"
      },
      "startsAt": "2024-11-21T07:34:23.708306749Z",
      "endsAt": "2024-11-21T07:39:23.708306749Z",
      "generatorURL": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
      "fingerprint": "eb584b7671528e35"
    }
  ],
  "groupLabels": {
    "alertname": "Test"
  },
  "commonLabels": {
    "alertname": "Test",
    "severity": "critical"
  },
  "commonAnnotations": {},
  "externalURL": "https://immortal-pig.in.signoz.cloud/alerts/edit?ruleId=1",
  "version": "4",
  "groupKey": "{}/{}:{alertname=\"Test\"}",
  "truncatedAlerts": 0
}

4.2.2 Manually Add/Edit the Filter

Trigger Filter (For Creating Incidents):
Payload JSON Key: "status"
Key Value Contains: [firing]
Map Incident With: "alerts".[0]."fingerprint"
This corresponds to the unique fingerprint from the SigNoz payload.
Incident Title From: "alerts".[0]."labels"."alertname"
This will use the event description as the incident title in Callgoose SQIBS.
Incident Description From: Leave this empty unless you want to use a specific key-value from the JSON payload. If a key is entered, only the value for that key will be used as the Incident Description instead of the full JSON. By default, the Incident Description will include the full JSON values.
Example: If you use the "alerts".[0]."annotations"."description" key in the Incident Description From field, the incident description will be the value of the "alerts".[0]."annotations"."description" key. In the example JSON payload provided earlier, this would result in a description like "This alert is fired when the defined metric (current value: 23.183333333333334) crosses the threshold (20)".
Resolve Filter (For Resolving Incidents):
Payload JSON Key: "status"
Key Value Contains: [resolved]
Incident Mapped With: "alerts".[0]."fingerprint"
This ensures the incident tied to the specific fingerprint is resolved when the alert status returns to normal.

Refer to the API Filter Instructions and FAQ for more details.

4.3 Finalizing Setup

Save the API Filters:
Ensure that the filters are correctly configured and saved in Callgoose SQIBS.
Double-check that all key mappings, incident titles, and descriptions are correctly aligned with the payload structure sent by SigNoz.

5. Testing and Validation

5.1 Triggering Alerts

Simulate a Monitoring Alert:
Trigger a condition in SigNoz that causes an alert (e.g., a high error rate on a monitored application).
Verify that an incident is created in Callgoose SQIBS with the correct information, including the incident title, description, and mapped values as defined in the API filters.

5.2 Resolving Alerts

Acknowledge and Resolve the Alert:
Once the issue is resolved in SigNoz (e.g., the error rate returns to normal levels), verify that the incident in Callgoose SQIBS is automatically marked as resolved.
Confirm that the resolution details are accurately reflected in Callgoose SQIBS, with the incident status updated accordingly.

6. Security Considerations

API Security: Ensure that the Callgoose SQIBS API endpoint is correctly configured and that the API token is securely stored and used.
SigNoz Permissions: Confirm that the webhook in SigNoz has appropriate permissions to send alerts and data to Callgoose SQIBS.
Data Encryption: Ensure that the transmission of data between SigNoz and Callgoose SQIBS is encrypted, especially if sensitive information is involved.

7. Troubleshooting

No Incident Created:
Verify that the webhook URL in SigNoz is correct and that the JSON payload structure matches the API filters configured in Callgoose SQIBS.
Check the API logs in Callgoose SQIBS to ensure that the payload is being received and processed correctly.
Incident Not Resolved:
Ensure that the resolve filter in Callgoose SQIBS is correctly configured and that the JSON payload sent by SigNoz matches the expected structure.
Use the debugging feature in Callgoose SQIBS to monitor and troubleshoot the incoming payloads.

8. Conclusion

This guide provides a comprehensive overview of how to integrate SigNoz with Callgoose SQIBS for effective incident management. By following the steps outlined, you can ensure that alerts from SigNoz are automatically reflected as incidents in Callgoose SQIBS, with proper resolution tracking when the issues are resolved.

For further customization or advanced use cases, refer to the official documentation for both SigNoz and Callgoose SQIBS:

This documentation will guide you through the integration process, ensuring that your incidents are managed effectively within Callgoose SQIBS based on real-time alerts from SigNoz.