Fiddler AI

Overview

This document provides a detailed guide to integrating Fiddler AI with Callgoose SQIBS for real-time Incident Management, Incident Auto Remediation, and Event-Driven Automation.

This integration enables automatic creation, updating, and resolution of incidents in Callgoose SQIBS based on alerts generated in Fiddler AI (e.g., Data Drift, Performance degradation, Data Integrity issues). The guide includes steps for creating webhooks in Fiddler AI, configuring alert rules, creating API filters in Callgoose SQIBS, and troubleshooting.

Prerequisites

• Fiddler AI account with Admin or Editor permissions to manage Integrations and Alert Rules.
• Callgoose SQIBS account with permissions to create API filters.
• Webhook/API Endpoint generated in Callgoose SQIBS.
• API Token from Callgoose SQIBS.

1. Obtain API Token and Endpoint Details

To integrate with Callgoose SQIBS, first obtain an API token and endpoint URL.

1.1 Generate an API Token

Follow the document: How to Create API Token in Callgoose SQIBS. Copy the generated token for later use.

1.2 Find the API Endpoint

Refer to Callgoose SQIBS API Endpoint Documentation. Your endpoint will follow the format:

https://****.callgoose.com/v1/process?from=FiddlerAI&token=xxxx

2. Configuring Fiddler AI to Send Alerts

Fiddler AI supports custom webhook integrations that can forward alert payloads to external systems like Callgoose SQIBS.

2.1 Install and Set Up Fiddler AI (Optional if already using)

2.1.1 Access Fiddler AI

1. Log in to your Fiddler AI instance.
2. Ensure you have the necessary roles to configure global integrations and project-level alerts.

2.2 Create a Webhook Destination

• Navigate to Settings (Gear Icon) $\rightarrow$ Integrations.
• Click + Add Integration.
• Select Webhook (sometimes labeled as "Other" or "Custom Webhook").
• Configure the Webhook:
• Integration Name: Callgoose SQIBS
• URL: Your API Endpoint (from Step 1.2).
• Method: POST
• Headers: (Optional) Leave blank unless you wish to pass the token via header instead of URL.
• Payload Template: Fiddler allows you to customize the JSON structure. It is recommended to use a structure that maps Fiddler variables (e.g., $model_name, $alert_name) to a clean JSON format.
• Click Test Integration to verify connectivity.
• Click Save.

2.3 Create a Monitor (Alert Policy)

1. Navigate to your Project and select the specific Model you wish to monitor.
2. Click on the Alerts tab.
3. Click + Add Alert.
4. Define the Alert Rule:

• Metric: Select the metric (e.g., Data Drift, Traffic, Performance).
• Condition: Set your comparison operator and threshold (e.g., Jensen-Shannon divergence > 0.1).
• Bin Size: Define the time window for evaluation.
• Under Notifications:
• Toggle the Integrations switch.
• Select the Callgoose SQIBS webhook created in Step 2.2.
• Click Save Alert.

2.4 Test the Webhook

Fiddler allows you to verify integrations during the setup phase.

• Go to Settings $\rightarrow$ Integrations.
• Locate the Callgoose SQIBS integration.
• Click Test.
• Verify that Callgoose SQIBS receives the test alert in the API Log.

3. Example JSON Payloads From Fiddler AI

Fiddler AI sends a JSON payload containing details about the model, the metric that breached the threshold, and the severity. The exact structure depends on the "Payload Template" configured in Step 2.2, but the following is the standard recommended format.

3.1 Triggered Alert Payload

{
  "alert_name": "Drift Detection - Production",
  "project_name": "Credit_Risk_Assessment",
  "model_name": "Credit_Score_v1",
  "severity": "CRITICAL",
  "alert_type": "Data Drift",
  "metric": "Jensen-Shannon Divergence",
  "value": 0.25,
  "threshold": 0.1,
  "timestamp": "2025-01-10T12:30:00Z",
  "url": "https://app.fiddler.ai/projects/...",
  "status": "triggered"
}

3.2 Resolved Alert Payload

Note: Fiddler AI primarily sends "Triggered" events. If your integration is configured to send resolution events (based on available Fiddler version features), the payload typically updates the status.

{
  "alert_name": "Drift Detection - Production",
  "project_name": "Credit_Risk_Assessment",
  "model_name": "Credit_Score_v1",
  "severity": "INFO",
  "timestamp": "2025-01-10T14:00:00Z",
  "status": "resolved"
}

4. Configuring Callgoose SQIBS

4.1 Create API Filters

API filters are required to map incoming alert payloads to incidents. You will use the data from the Fiddler JSON payload to populate incident fields.

4.1.1 Trigger Filter (For Creating Incidents)

• Payload JSON Key: "status" (or severity)
• Key Value Contains: triggered (or CRITICAL)
• Map Incident With: "model_name" (This ensures all alerts for this model are grouped).
• Incident Title From: "alert_name"
• Incident Description From: Leave empty to recieve full payload as description.

4.1.2 Resolve Filter (For Resolving Incidents)

• Payload JSON Key: "status"
• Key Value Contains: resolved
• Incident Mapped With: "model_name" (Must match the key used in the Trigger Filter).

5. Testing and Validation

5.1 Triggering Alerts

• In Fiddler AI, you can lower the threshold of an existing alert rule temporarily (e.g., set Drift threshold to 0.0001).
• Send a sample payload or wait for the next bin calculation.
• Confirm a new incident appears in Callgoose SQIBS with the correct Model Name and Severity.

5.2 Resolving Alerts

• Revert the threshold in Fiddler AI to its original value.
• Once the metric stabilizes and Fiddler evaluates the next bin as "Healthy," a resolution webhook is sent (if configured).
• Confirm the incident in Callgoose SQIBS auto-resolves.

6. Debugging and Troubleshooting

• Enable Debug in the Callgoose SQIBS API Token settings.
• Review logs under API Log in Callgoose SQIBS.
• Common issues:
• No incident created: Verify the status or severity field in the Fiddler payload matches the "Key Value Contains" in your Callgoose Trigger Filter.
• 401 Unauthorized: Ensure the API Token in the URL is correct.
• Empty Fields: Check if your Fiddler "Payload Template" correctly maps the internal Fiddler variables to the JSON keys expected by Callgoose.

7. Security Considerations

• Keep your Callgoose API token private.
• Ensure the Webhook URL uses HTTPS.
• In Fiddler, use the "Test" function to ensure data is transmitted securely before saving.
• Rotate API tokens periodically in Callgoose SQIBS.

8. Conclusion

By following this guide, you can integrate Fiddler AI with Callgoose SQIBS for seamless incident creation and resolution regarding your ML Models. This ensures your MLOps teams have real-time visibility into Model Drift, Integrity, and Performance issues directly within their incident management workflow.

For further customization or advanced use cases, refer to the official documentation for both Fiddler AI and Callgoose SQIBS:

• Fiddler AI Documentation
• Callgoose SQIBS API Token Documentation
• Callgoose SQIBS API Endpoint Documentation
• API Filter Instructions and FAQ
• How to Send API