Soda

Overview

This guide provides a detailed walkthrough for integrating Soda (Soda Cloud/Soda Library) with Callgoose SQIBS. By connecting these platforms, you can transform data quality checks into real-time incidents.

The integration utilizes Soda Cloud Webhooks to push data contract violations and check failures directly to Callgoose. Using API Filters, Callgoose can automatically create incidents when data quality drops and resolve them once the data is back within expected parameters.

Prerequisites

Before beginning, ensure you have the following:

• A Callgoose SQIBS account with permissions to create API Filters and access the integration endpoint.
• A Soda Cloud account with Admin or Manager access to configure Webhooks.
• A valid Callgoose API token and Api endpoint URL.
• A Soda Check (sodacl) configured to run on a schedule or via a CI/CD pipeline.

1. Prepare Callgoose: Obtain Endpoint and Token

1.1 Retrieve Callgoose API Endpoint

Generate or locate your unique Callgoose process endpoint:

https://****.callgoose.com/v1/process?from=soda&token=xxxx

store the API token securely.Do not expose it in public repositories or logs.

2. Configure Soda Cloud Webhook

store the API token securely.Do not expose it in public repositories or logs

2.1 Access Webhook Settings

In Soda Cloud, webhooks are managed centrally to notify external systems of check results.

1. Log in to Soda Cloud.
2. Navigate to Settings → Webhooks.
3. Click Create Webhook (or the + icon).

2.2 Configure Webhook Fields

Set the following options to ensure data flows correctly to Callgoose:

• Webhook Name: Callgoose SQIBS Integration
• Webhook URL: Paste the Callgoose endpoint from Step 1.
• Request Method: POST
• Events to Trigger: * Select Check Fail (for incident creation).
• Select Check Pass (for auto-resolution).
• Payload Format: JSON

2.3 Custom Headers (Optional)

If you require additional security, you can add a custom header:

• Authorization: Bearer <Your_Callgoose_Token>

3. Example Soda Payloads

Soda sends detailed metadata about the failed check. Below are simplified examples for mapping.

Firing State (Check Fail)

JSON

{
  "action": "check_failed",
  "check_id": "soda-uuid-12345",
  "check_name": "Check row count > 0",
  "dataset": "orders_table",
  "severity": "fail",
  "check_value": 0,
  "incident_id": "98765"
}

Resolved State (Check Pass)

JSON

{
  "action": "check_passed",
  "check_id": "soda-uuid-12345",
  "check_name": "Check row count > 0",
  "dataset": "orders_table"
}

4. Create API Filters in Callgoose SQIBS

Callgoose API Filters interpret the Soda JSON payload to manage the incident lifecycle.

4.1 Trigger Filter — Create Incident

Use the following configuration to automatically create an incident in Callgoose SQIBS when a Soda check fails.

Configuration

• Payload JSON Key: action
• Key Value Contains: check_failed
• Map Incident With: check_id
• Incident Title: check_name
• Incident Description: dataset

4.2 Resolve Filter — Auto-Resolve Incident

Use the following configuration to automatically resolve an incident when the Soda check passes again.

Configuration

• Payload JSON Key: action
• Key Value Contains: check_passed
• Incident Mapped With: check_id

Refer to the API Filter Instructions and FAQ for more details.

5. Verify and Test the Integration

5.1 Test Initial Delivery

1. In Soda Cloud, use the Test Webhook button.
2. Check the Callgoose API Logs to ensure the "Test" payload arrived successfully.

5.2 Final Verification

1. Trigger Failure: Run a Soda scan on a dataset you know will fail. Verify that a Callgoose incident is created.
2. Verify Resolution: Fix the data or the check criteria and re-run the Soda scan. Confirm the Callgoose incident status changes to Resolved.

6. Troubleshooting

• No Incidents Created: Ensure the check_id is consistent in the payload. Check if the Soda Webhook is set to trigger on "Warning" vs "Fail"—Callgoose needs to match the specific action string.
• Authentication Errors: Double-check that the token parameter in your Callgoose URL is correct and hasn't expired.
• Payload Format Changes: Soda updates its Cloud API periodically. If fields are missing, inspect the raw JSON in the Callgoose API Request Logs and update your Filter mappings accordingly.

7. Conclusion

By integrating Soda with Callgoose SQIBS, you move from reactive data fixing to proactive data reliability. This loop ensures that data engineering teams are alerted the moment a data contract is broken and notified automatically when data integrity is restored.

• soda documentation
• Callgoose SQIBS API Token Documentation
• Callgoose SQIBS API Endpoint Documentation
• API Filter Instructions and FAQ
• How to Send API