Chronosphere

Overview

This document provides a detailed guide to integrating Chronosphere with Callgoose SQIBS for real-time Incident Management, Incident Auto Remediation, and Event-Driven Automation.

This integration enables automatic creation, updating, and resolution of incidents in Callgoose SQIBS based on alerts generated in Chronosphere. The guide includes steps for creating monitors (alert policies) in Chronosphere, configuring webhook notifications, creating API filters in Callgoose SQIBS, and troubleshooting.

Prerequisites

• Chronosphere account with permissions to manage Monitors and Notification Policies.
• Callgoose SQIBS account with permissions to create API filters.
• Webhook/API Endpoint generated in Callgoose SQIBS.
• API Token from Callgoose SQIBS.

1. Obtain API Token and Endpoint Details

To integrate with Callgoose SQIBS, first obtain an API token and endpoint URL.

1.1 Generate an API Token

Follow the document: How to Create API Token in Callgoose SQIBS. Copy the generated token for later use.

1.2 Find the API Endpoint

Refer to Callgoose SQIBS API Endpoint Documentation. Your endpoint will follow the format: https://****.callgoose.com/v1/process?from=Chronosphere&token=xxxx

2. Configuring Chronosphere to Send Alerts

Chronosphere supports notification policies with webhook-based notifications (often utilizing standard Alertmanager webhook formats). These webhooks forward alert payloads to Callgoose SQIBS.

2.1 Install & Set Up Chronosphere (Optional if already using)

2.1.1 Access Chronosphere

• Log in to your Chronosphere tenant.
• Ensure you have the necessary roles (Admin or Editor) to configure alerts.

2.2 Create a Webhook Destination

• Navigate to Alerting → Notification Policies (or Notifier Configuration depending on your version).
• Select Webhooks or New Notifier.
• Configure the Webhook:
• Name: Callgoose SQIBS
• Endpoint URL: Your API Endpoint (from Step 1.2).
• HTTP Method: POST
• Authentication: None (Authentication is handled via the token parameter in the URL).
• 4. Save the configuration.

2.3 Create a Monitor (Alert Policy)

• Go to Alerting → Monitors.
• Click Create Monitor.
• Define the Monitor:
• Query: Enter your PromQL query (e.g., http_request_duration_seconds > 0.5).
• Conditions: Set the duration and threshold.
• Labels/Annotations: Add specific descriptions or summaries.
• Under Notifications:
• Select the Callgoose SQIBS webhook created in Step 2.2.
• Save the Monitor.

2.4 Test the Webhook

Chronosphere allows you to verify notifiers.

• Go to the Notifier/Webhook settings.
• Click Test or Send Test Notification.
• Verify that Callgoose SQIBS receives the test alert in the logs.

3. Example JSON Payloads From Chronosphere

Chronosphere generally sends payloads in the Prometheus Alertmanager format. These payloads are representative of what Chronosphere sends.

3.1 Triggered Alert Payload

JSON

{
  "receiver": "Callgoose SQIBS",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "HighLatency",
        "severity": "critical",
        "service": "payment-gateway"
      },
      "annotations": {
        "summary": "High Latency detected",
        "description": "p99 latency exceeded 500ms"
      },
      "startsAt": "2025-01-10T12:30:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "fingerprint": "abc123456789"
    }
  ],
  "groupLabels": {
    "alertname": "HighLatency"
  },
  "commonLabels": {
    "alertname": "HighLatency",
    "severity": "critical"
  },
  "commonAnnotations": {
    "summary": "High Latency detected"
  },
  "externalURL": "https://chronosphere.io/alerts/123",
  "version": "4",
  "groupKey": "{}:{alertname=\"HighLatency\"}"
}

3.2 Resolved Alert Payload

JSON

{
  "receiver": "Callgoose SQIBS",
  "status": "resolved",
  "alerts": [
    {
      "status": "resolved",
      "labels": {
        "alertname": "HighLatency",
        "severity": "critical",
        "service": "payment-gateway"
      },
      "annotations": {
        "summary": "Latency normal",
        "description": "p99 latency back to normal levels"
      },
      "startsAt": "2025-01-10T12:30:00Z",
      "endsAt": "2025-01-10T12:42:00Z",
      "fingerprint": "abc123456789"
    }
  ],
  "groupLabels": {
    "alertname": "HighLatency"
  },
  "commonLabels": {
    "alertname": "HighLatency",
    "severity": "critical"
  },
  "commonAnnotations": {
    "summary": "High Latency detected"
  },
  "externalURL": "https://chronosphere.io/alerts/123",
  "version": "4",
  "groupKey": "{}:{alertname=\"HighLatency\"}"
}

4. Configuring Callgoose SQIBS

4.1 Create API Filters

API filters are required to map incoming alert payloads to incidents. Chronosphere payloads rely on the groupKey or fingerprint for unique identification and commonAnnotations for titles.

4.1.1 Trigger Filter (For Creating Incidents)

• Payload JSON Key: status
• Key Value Contains: firing
• Map Incident With: groupKey (or alerts[0].fingerprint if specific alert mapping is required)
• Incident Title From: commonAnnotations.summary (or alerts[0].annotations.summary)
• Incident Description From: commonAnnotations.description (optional)
• Leave empty to use full JSON payload.

4.1.2 Resolve Filter (For Resolving Incidents)

• Payload JSON Key: status
• Key Value Contains: resolved
• Incident Mapped With: groupKey (Must match the key used in the Trigger Filter)

5. Testing & Validation

5.1 Triggering Alerts

1. Trigger a failure or modify a Monitor threshold in Chronosphere to force an alert.
2. Confirm a new incident appears in Callgoose SQIBS.

5.2 Resolving Alerts

1. Revert the condition or metric in Chronosphere to normal.
2. Chronosphere sends a resolved payload.
3. Confirm the incident in Callgoose SQIBS auto-resolves.

6. Debugging & Troubleshooting

1. Enable Debug in the API Token settings.
2. Review logs under API Log in Callgoose SQIBS.
3. Common issues:

• No incident created: Incorrect webhook URL or API filter mapping (ensure status matches firing).
• Incident not resolved: Mismatched groupKey or fingerprint between the Trigger and Resolve filters.
• Payload not received: Verify firewall settings or Chronosphere webhook configuration.

7. Security Considerations

• Keep API token private.
• Use HTTPS endpoints only.
• Restrict webhook access where possible.
• Rotate API tokens periodically.

8. Conclusion

By following this guide, you can integrate Chronosphere with Callgoose SQIBS for seamless incident creation, updates, and resolution. This ensures your DevOps and SRE teams have real-time visibility into system performance and alerts.

For additional reference:

• Chronosphere Documentation
• Callgoose SQIBS API Token Documentation
• Callgoose SQIBS API Endpoint Documentation
• API Filter Instructions and FAQ
• How to Send API