Montecarlodata

Overview

This document provides a detailed guide to integrating Monte Carlo Data with Callgoose SQIBS for real-time Data Reliability and Incident Management.

By connecting Monte Carlo’s data observability alerts (Schema changes, Freshness, Volume, and Monitor failures) with Callgoose, data engineering teams can automate incident creation, manage on-call rotations for data outages, and ensure automated incident resolution when data monitors return to a healthy state.

Prerequisites

Before beginning, ensure you have the following:

• A Callgoose SQIBS account with permissions to create API Filters and access the integration endpoint.
• A Monte Carlo account with "Owner" or "Editor" privileges to configure Notifications.
• A valid Callgoose API token and Api endpoint URL.
• An active Monte Carlo Monitor (e.g., a Freshness or Custom SQL monitor) to generate test alerts.

1. Prepare Callgoose: Obtain Endpoint and Token

1.1 Retrieve Callgoose API Endpoint

Generate or locate your Callgoose process endpoint:

https://****.callgoose.com/v1/process?from=montecarlodata&token=xxxx

Store the API token securely and never commit it to source control.

2. Configure Monte Carlo Webhook

2.1 Choose the Webhook Scope

Monte Carlo allows you to send alerts via webhooks at a global level or filtered by specific audiences/tags.

1. Log in to your Monte Carlo dashboard.
2. Navigate to Settings → Notifications.
3. Click Add Integration and select Webhook.

2.2 Configure Webhook Fields

Set the following options:

• Webhook Name: Callgoose SQIBS
• Webhook URL: Paste the Callgoose endpoint from Step 1.
• Authorization Header (Optional): If you require additional headers, configure them here; otherwise, the token in the URL is sufficient.
• Resolution Notifications: Ensure "Send notifications when an incident is resolved" is checked. This is critical for auto-resolution in Callgoose.

2.3 Verification

Once saved, use the "Send Test" button in Monte Carlo to verify that the webhook reaches Callgoose successfully.

3. Example Monte Carlo Payloads

3.1 Trigger Payload (Incident Created/Firing)

Monte Carlo sends a JSON payload when a monitor fails:

JSON

{
  "event_type": "incident_created",
  "incident_id": "mc-data-99812",
  "incident_type": "Freshness",
  "table_name": "analytics.orders",
  "description": "Table hasn't been updated in 6 hours",
  "severity": "high",
  "url": "https://getmontecarlo.com/incidents/mc-data-99812"
}

3.2 Resolution Payload (Incident Resolved)

When a user resolves the incident in Monte Carlo or the monitor recovers:

JSON

{
  "event_type": "incident_resolved",
  "incident_id": "mc-data-99812",
  "resolved_at": "2025-10-01T14:00:00Z"
}

4. Create API Filters in Callgoose SQIBS

Callgoose API Filters convert incoming Monte Carlo webhooks into actionable incidents.

4.1 Trigger Filter — Create Incident

Configure this filter to catch new Monte Carlo incidents:

• Payload JSON Key: event_type
• Key Value Contains: incident_created
• Map Incident With: incident_id
• Incident Title: incident_type on table_name
• Incident Description: description

4.2 Resolve Filter — Auto-Resolve Incident

Configure this filter to close the incident automatically:

• Payload JSON Key: event_type
• Key Value Contains: incident_resolved
• Incident Mapped With: incident_id

Refer to the API Filter Instructions and FAQ for more details.

5. Verify and Test the Integration

5.1 Test Initial Delivery

1. Trigger a "Test Notification" from the Monte Carlo Notification settings.
2. Check the API Logs in Callgoose SQIBS to confirm the payload was received.
3. Identify the exact JSON keys (Monte Carlo keys can occasionally vary based on monitor type).

5.2 Final Verification

1. Trigger: Manually trip a monitor in Monte Carlo (or wait for a scheduled failure) → Callgoose incident should be created.
2. Resolve: Mark the incident as "Fixed" or "Resolved" in Monte Carlo → Callgoose incident should be auto-resolved.

6. Troubleshooting

• Payload Not Received: Check the "Webhook History" in Monte Carlo Settings. Ensure the Callgoose token hasn't expired.
• Incidents Not Resolving: Verify that the incident_id in the resolution payload matches the incident_id from the trigger payload. If Monte Carlo sends a different ID format for resolution, update the mapping in Callgoose.
• Missing Table Info: If specific fields like table_name are missing, check if the alert is coming from a "Custom SQL" monitor vs. an "Automated Freshness" monitor, as schemas may differ.

7. Conclusion

Integrating Monte Carlo with Callgoose SQIBS ensures that data downtime is treated with the same urgency as software downtime. By automating the flow from data anomaly detection to on-call notification, teams can significantly reduce Time to Detection (TTD) and Time to Resolution (TTR) for critical data assets.

For further customization or advanced use cases, refer to:

• montecarlodata documentation
• Callgoose SQIBS API Token Documentation
• Callgoose SQIBS API Endpoint Documentation
• API Filter Instructions and FAQ
• How to Send API