Honeycomb

Overview

This document offers a comprehensive guide for integrating Honeycomb with Callgoose SQIBS to support real-time Incident Management, Incident Auto Remediation, Event-Driven Automation, and various automation processes. This integration allows for the automatic creation, updating, and resolution of incidents in Callgoose SQIBS, triggered by alerts from Honeycomb. The guide covers the steps for setting up triggers in Honeycomb, configuring Webhook notifications, creating API filters in Callgoose SQIBS, and troubleshooting.

1. Obtain API Token and Endpoint Details

To integrate with Callgoose SQIBS, you first need to obtain an API token and find the API endpoint details.

• Generate an API Token:
• Follow the guide on How to Create API Token in Callgoose SQIBS.
• Find the API Endpoint:
• Refer to the Callgoose SQIBS API Endpoint Documentation to get the endpoint details where the JSON payloads from Honeycomb will be sent.

2. Debugging and Troubleshooting

You can enable debugging in the API tokens used with Honeycomb notifications for troubleshooting purposes.

• Enable Debugging:
• You can update the debug value when adding or updating an API token.
• When API tracking is enabled, logs are stored in the API log section for your review. The debugging option will automatically disable after 48 hours.
• When API tracking is turned off, no logs are saved in the API log.
• Using API Log for Troubleshooting:
• The API log provides detailed information on all API calls made to Callgoose SQIBS.
• You can check the JSON values in each API log entry for troubleshooting purposes.
• Use the information in the API log to create or refine API filters to ensure incidents are created correctly based on the API payloads received.
• Callgoose SQIBS creates incidents according to your API filter configuration, giving you full control over how alerts from different services trigger incidents and alerts for your support team or automation processes.

3. Configuring Honeycomb to Send JSON Payloads

To configure Honeycomb to generate the JSON payloads, follow the steps outlined below. These steps will guide you through setting up the necessary triggers and webhook notifications within Honeycomb to ensure that the JSON payloads match those expected by Callgoose SQIBS.

3.1 Setting Up Triggers in Honeycomb

To generate the required JSON payloads, you first need to set up triggers within Honeycomb. Before creating a trigger, ensure that data is collected in Honeycomb from the connected application. This data is essential for creating new trigger.

• Log in to the Honeycomb Console:
• Access the Honeycomb platform using your account credentials.
• Create team.
• Send data to Honeycomb:
• Navigate to the Home section in Honeycomb.
• Use the appropriate Honeycomb API to instrument your application. This will allow your application to send events and traces to Honeycomb.
• Follow the detailed instructions in the Send Data to Honeycomb for integrating your specific technology stack (e.g., JavaScript, Python, Go, etc.).
• Navigate to the Integration Center:
• In the Honeycomb console, follow these steps to access the Integration Center:
• Click on Environment from the homepage of honeycomb.
• Select Settings.
• Navigate to API Keys.
• Click on Management API Keys.
• Go to Integrations.
• Click on Add Integration.
• Configure the Webhook Integration:
• In the Provider section locate and select Webhook.
• Provide a name.
• In the Webhook URL section add Webhook URL provided by callgoose SQIBS. Refer API endpoint for more information on the format of URL.
• Type CallgooseSQIBS as shared secret key.
• Navigate to the Triggers Section:
• In Honeycomb, go to Triggers section.
• Create a New Trigger:
• Click on New Trigger to create a new trigger.
• Specify Trigger Conditions: Define the conditions that will activate the trigger, such as specific performance metrics, security events, or thresholds.
• Configure the Recipents: Click on Add Recipient and choose Webhook as the Recipient. Click on Add.
• Save the Trigger.
• Refer Honeycomb Triggers for more details.
• Test and Verify Payload:
• Honeycomb does not offer a direct option to customize the payload format. Instead you can click on test after creating trigger. Then the payload can be viewed from API log of Callgoose SQIBS.
• Refer to the API log in Callgoose SQIBS to review the payload structure and ensure it includes key fields like "status", "id", "description", "alert_type".

4. Configuring Callgoose SQIBS

4.1 Create API Filters in Callgoose SQIBS

To correctly map incidents from the Honeycomb triggers, you need to create API filters based on the JSON payloads received.

4.1.1 Example JSON Payloads from Honeycomb

Alert Triggered (status: "TRIGGERED")

json

{
  "version": "v0.1.0",
  "name": "Test",
  "id": "mH9UsEVi4W2",
  "trigger_description": "test",
  "status": "TRIGGERED",
  "alert_type": "on_change",
  "summary": "TRIGGER TEST: Triggered: Test",
  "description": "notification environment:\nCurrent value greater than threshold value (0)",
  "operator": "greater than",
  "threshold": 0,
  "result_url": "https://ui.honeycomb.io/***/environments/notification/datasets/callgoose/result/****/a/****?utm_content=view_graph&utm_medium=Trigger&utm_source=webhook",
  "result_groups": [
    {
      "Group": {
        "duration_ms": null
      },
      "Result": 0
    }
  ],
  "result_groups_triggered": [],
  "trigger_url": "https://ui.honeycomb.io/****/environments/notification/datasets/callgoose/triggers/****?utm_content=edit_trigger&utm_medium=Trigger&utm_source=webhook",
  "is_test": true
}

Alert Resolved (status: "OK")

json

{
  "version": "v0.1.0",
  "name": "Test",
  "id": "mH9UsEVi4W2",
  "trigger_description": "test",
  "status": "OK",
  "alert_type": "on_change",
  "summary": "TRIGGER TEST: Resolved: Test",
  "description": "notification environment:\nCurrent value no longer greater than threshold value (0) for any duration_ms",
  "operator": "greater than",
  "threshold": 0,
  "result_url": "https://ui.honeycomb.io/****/environments/notification/datasets/callgoose/result/****/a/****?utm_content=view_graph&utm_medium=Trigger&utm_source=webhook",
  "result_groups": [
    {
      "Group": {
        "duration_ms": null
      },
      "Result": 0
    }
  ],
  "result_groups_triggered": [],
  "trigger_url": "https://ui.honeycomb.io/****/environments/notification/datasets/callgoose/triggers/****?utm_content=edit_trigger&utm_medium=Trigger&utm_source=webhook",
  "is_test": true
}

4.2 Configuring API Filters

4.2.1 Integration Templates

If you see a Honeycomb integration template in the "Select Integration Template" dropdown in the API filter settings, you can use it to automatically add the necessary Trigger and Resolve filters along with other values. The values added by the template can be modified to customize the integration according to your requirements.

4.2.2 Manually Add/Edit the Filter

• Trigger Filter (For Creating Incidents):
• Payload JSON Key: "status"
• Key Value Contains: [TRIGGERED]
• Map Incident With: "id"
• This corresponds to the unique id from the Honeycomb payload.
• Incident Title From: "summary"
• This will use the event summary as the incident title in Callgoose SQIBS.
• Incident Description From: Leave this empty unless you want to use a specific key-value from the JSON payload. If a key is entered, only the value for that key will be used as the Incident Description instead of the full JSON. By default, the Incident Description will include the full JSON values.
• Example: If you use the "description" key in the Incident Description From field, the incident description will be the value of the "description" key. In the example JSON payload provided earlier, this would result in a description like "notification environment:\nCurrent value greater than threshold value (0)".
• Resolve Filter (For Resolving Incidents):
• Payload JSON Key: "status"
• Key Value Contains: [OK]
• Incident Mapped With: "id"
• This ensures the incident tied to the specific event id is resolved when the alert status returns to normal.

Refer to the API Filter Instructions and FAQ for more details.

4.3 Finalizing Setup

• Save the API Filters:
• Ensure that the filters are correctly configured and saved in Callgoose SQIBS.
• Double-check that all key mappings, incident titles, and descriptions are correctly aligned with the payload structure sent by Honeycomb.

5. Testing and Validation

5.1 Triggering Alerts

• Simulate a Monitoring Alert:
• Trigger a condition in Honeycomb that causes an alert (e.g., high latency on a monitored application).
• Verify that an incident is created in Callgoose SQIBS with the correct information, including the incident title, description, and mapped values as defined in the API filters.

5.2 Resolving Alerts

• Acknowledge and Resolve the Alert:
• Once the issue is resolved in Honeycomb (e.g., latency returns to normal levels), verify that the incident in Callgoose SQIBS is automatically marked as resolved.
• Confirm that the resolution details are accurately reflected in Callgoose SQIBS, with the incident status updated accordingly.

6. Security Considerations

• API Security: Ensure that the Callgoose SQIBS API endpoint is correctly configured and that the API token is securely stored and used.
• Honeycomb Permissions: Confirm that the Webhook in Honeycomb has appropriate permissions to send alerts and data to Callgoose SQIBS.

7. Troubleshooting

• No Incident Created:
• Verify that the Webhook URL in Honeycomb is correct and that the JSON payload structure matches the API filters configured in Callgoose SQIBS.
• Check the API logs in Callgoose SQIBS to ensure that the payload is being received and processed correctly.
• Incident Not Resolved:
• Ensure that the resolve filter in Callgoose SQIBS is correctly configured and that the JSON payload sent by Honeycomb matches the expected structure.
• Use the debugging feature in Callgoose SQIBS to monitor and troubleshoot the incoming payloads.

8. Conclusion

This guide provides a comprehensive overview of how to integrate Honeycomb with Callgoose SQIBS for effective incident management. By following the steps outlined, you can ensure that alerts from Honeycomb are automatically reflected as incidents in Callgoose SQIBS, with proper resolution tracking when the issues are resolved.

For further customization or advanced use cases, refer to the official documentation for both Honeycomb and Callgoose SQIBS:

• Honeycomb Documentation
• Callgoose SQIBS API Token Documentation
• Callgoose SQIBS API Endpoint Documentation
• API Filter Instructions and FAQ
• How to Send API

This documentation will guide you through the integration process, ensuring that your incidents are managed effectively within Callgoose SQIBS based on real-time alerts from Honeycomb.