Buildkite

Overview

This document provides a detailed guide to integrating Buildkite Pipelines with Callgoose SQIBS for real-time Incident Management, Incident Auto Remediation, Event-Driven Automation, and other automation purposes. The integration enables automatic creation, updating, and resolution of incidents in Callgoose SQIBS based on events triggered in Buildkite Pipelines. The guide includes steps for setting up webhooks in Buildkite, configuring webhook notifications, creating API filters in Callgoose SQIBS, and troubleshooting.

Prerequisites

1. Obtain API Token and Endpoint Details

To integrate with Callgoose SQIBS, you first need to obtain an API token and find the API endpoint details.

• Generate an API Token:
• Follow the guide on How to Create API Token in Callgoose SQIBS.
• Find the API Endpoint:
• Refer to the Callgoose SQIBS API Endpoint Documentation to get the endpoint details where the JSON payloads from Buildkite will be sent.

2. Debugging and Troubleshooting

You can enable debugging in the API tokens used with Buildkite webhooks for troubleshooting purposes.

• Enable Debugging:
• You can update the debug value when adding or updating an API token.
• When API tracking is enabled, logs are stored in the API log section for your review. The debugging option will automatically disable after 48 hours.
• When API tracking is turned off, no logs are saved in the API log.
• Using API Log for Troubleshooting:
• The API log provides detailed information on all API calls made to Callgoose SQIBS.
• You can check the JSON values in each API log entry for troubleshooting purposes.
• Use the information in the API log to create or refine API filters to ensure incidents are created correctly based on the API payloads received.
• Callgoose SQIBS creates incidents according to your API filter configuration, giving you full control over how alerts from different services trigger incidents and alerts for your support team or automation processes.

3. Configuring Buildkite Pipelines to Send JSON Payloads

3.1 Setting Up Webhooks in Buildkite Pipelines

To generate the required JSON payloads, you first need to set up webhooks within Buildkite.

• Log in to the Buildkite Dashboard:
• Access the Buildkite platform using your account credentials.
• Navigate to the Webhooks Section:
• In the Buildkite dashboard, go to Settings > Webhooks.
• Create a New Webhook:
• Click on Add Webhook to create a new webhook.
• Specify the events that will trigger the webhook, such as build status changes, job failures, or pipeline completions.
• Configure the Webhook URL:
• Enter the webhook URL provided by Callgoose SQIBS in the URL field.
• Ensure the protocol is set to HTTPS for secure communication.

3.2 Configuring the Webhook Notification

To ensure that the JSON payload sent matches the examples provided, follow these steps when configuring the webhook:

• Add Webhook URL:
• In the Webhook URL field, enter the endpoint provided by Callgoose SQIBS.
• Ensure the protocol is HTTPS for secure data transmission.
• Customize Payload Format:
• Ensure that the payload includes key fields like "status", "event", "severity", "description", "timestamp", and others as shown in the example payloads.
• Example Payload Setup:

json

{
  "event": {
    "id": "$EVENT_ID",
    "status": "$STATUS",
    "severity": "$SEVERITY",
    "description": "$DESCRIPTION",
    "host": "$HOST",
    "timestamp": "$TIMESTAMP"
  }
}

• Placeholder Explanation:
• $STATUS: Replaces with the status of the event (e.g., OK, CRITICAL).
• $EVENT_ID: A unique identifier for the event.
• $SEVERITY: The level of urgency.
• $DESCRIPTION: A descriptive message of the event.
• $HOST: The affected host.
• $TIMESTAMP: The time the event occurred.
• Test the Webhook Configuration:
• Before activating the webhook, perform a test to ensure that the JSON payload is correctly formatted and is being sent to the Callgoose SQIBS API endpoint as expected.
• Review the payload in Callgoose SQIBS to confirm that it matches the expected structure.

4. Configuring Callgoose SQIBS

4.1 Create API Filters in Callgoose SQIBS

To correctly map incidents from Buildkite events, you need to create API filters based on the JSON payloads received.

4.1.1 Example JSON Payloads from Buildkite Pipelines

Event Triggered (status: "FAILED")

json

{
  "event": {
    "id": "event123",
    "status": "FAILED",
    "severity": "high",
    "description": "Pipeline build failed",
    "host": "pipeline1.buildkite.com",
    "timestamp": "2024-08-05T12:00:00.000Z"
  }
}

Event Resolved (status: "PASSED")

json

{
  "event": {
    "id": "event123",
    "status": "PASSED",
    "severity": "low",
    "description": "Pipeline build passed successfully",
    "host": "pipeline1.buildkite.com",
    "timestamp": "2024-08-05T12:30:00.000Z"
  }
}

4.2 Configuring API Filters

4.2.1 Integration Templates

If you see a Buildkite Pipelines integration template in the "Select Integration Template" dropdown within the API filter settings, you can use it to automatically add the necessary Trigger and Resolve filters along with other values. The values added by the template can be modified to customize the integration according to your requirements.

4.2.2 Manually Add/Edit the Filter

Trigger Filter (For Creating Incidents):

• Payload JSON Key: "status"
• Key Value Contains: [FAILED, ERROR]
• Map Incident With: "event.id"
• This corresponds to the unique event.id from the Buildkite payload.
• Incident Title From: "event.description"
• This uses the event description as the incident title in Callgoose SQIBS.
• Incident Description From: Leave this empty unless you want to use a specific key-value from the JSON payload. If a key is entered, only the value for that key will be used as the Incident Description instead of the full JSON. By default, the Incident Description will include the full JSON values.

Resolve Filter (For Resolving Incidents):

• Payload JSON Key: "status"
• Key Value Contains: [PASSED, FIXED]
• Incident Mapped With: "event.id"
• This ensures the incident tied to the specific event.id is resolved when the event status returns to normal.

Refer to the API Filter Instructions and FAQ for more details.

4.3 Finalizing Setup

• Save the API Filters:
• Ensure that the filters are correctly configured and saved in Callgoose SQIBS.
• Double-check that all key mappings, incident titles, and descriptions are correctly aligned with the payload structure sent by Buildkite.
• Test the Integration:
• Manually trigger an event in Buildkite to test if incidents are created in Callgoose SQIBS.
• Verify that the incident appears in Callgoose SQIBS with the correct title, description, and mapped values.
• Resolve the event in Buildkite and ensure that the corresponding incident in Callgoose SQIBS is marked as resolved.

5. Testing and Validation

5.1 Triggering Events

• Simulate a Pipeline Event:
• Trigger a condition in Buildkite that causes an event (e.g., pipeline build failure).
• Verify that an incident is created in Callgoose SQIBS with the correct information.

5.2 Resolving Events

• Acknowledge and Resolve the Event:
• Once the issue is resolved in Buildkite (e.g., pipeline build success), verify that the incident in Callgoose SQIBS is automatically marked as resolved.

6. Security Considerations

• API Security: Ensure that the Callgoose SQIBS API endpoint is correctly configured and that the API token is securely stored and used. API tokens should be treated as sensitive information, and access should be restricted to authorized personnel only.
• Buildkite Permissions: Confirm that the webhook in Buildkite has the appropriate permissions to send alerts and data to Callgoose SQIBS. Limit the scope of permissions to only what is necessary to avoid unauthorized access or data leakage.
• Data Encryption: Ensure that the transmission of data between Buildkite and Callgoose SQIBS is encrypted using HTTPS. This will help protect the integrity and confidentiality of the data being transmitted, especially if sensitive information is included in the payloads.
• Audit Logs: Regularly review the audit logs in both Buildkite and Callgoose SQIBS to monitor for any unauthorized access or anomalies in the integration setup. This helps in quickly identifying and responding to potential security incidents.
• Incident Response Plan: Implement an incident response plan that includes steps to be taken in the event of a security breach or unauthorized access to the integration between Buildkite and Callgoose SQIBS.

7. Troubleshooting

• No Incident Created: If no incident is created in Callgoose SQIBS, verify that the webhook URL in Buildkite is correct and that the JSON payload structure matches the API filters configured in Callgoose SQIBS. Check the API logs in Callgoose SQIBS for any errors or discrepancies in the incoming payload.
• Incident Not Resolved: Ensure that the resolve filter in Callgoose SQIBS is correctly configured and that the JSON payload sent by Buildkite contains the expected status update to trigger the resolution of the incident. Verify that the correct keys and values are being mapped in the API filter settings.
• Webhook Test Failures: If the webhook test fails, double-check the webhook URL, API token, and payload format. Make sure that the payload is being sent as expected and that there are no network issues preventing communication between Buildkite and Callgoose SQIBS.
• Debugging API Calls: Use the debugging option in Callgoose SQIBS to track and review API calls made by Buildkite. This will help identify any issues with the payload structure or content that may be causing problems in the incident creation or resolution process.

8. Conclusion

This guide provides a comprehensive overview of how to integrate Buildkite Pipelines with Callgoose SQIBS for effective incident management. By following the steps outlined, you can ensure that events from Buildkite are automatically reflected as incidents in Callgoose SQIBS, with proper resolution tracking when the issues are resolved.

For further customization or advanced use cases, refer to the official documentation for both Buildkite and Callgoose SQIBS:

• Buildkite Documentation
• Callgoose SQIBS API Token Documentation
• Callgoose SQIBS API Endpoint Documentation
• API Filter Instructions and FAQ
• How to Send API

This documentation will guide you through the integration process, ensuring that your incidents are managed effectively within Callgoose SQIBS based on real-time events from Buildkite Pipelines.