Integrations
LangSmith
Overview
This document provides a detailed guide to integrating Langsmith with Callgoose SQIBS for real-time Incident Management, Incident Auto Remediation, and Event-Driven Automation.
This integration enables automatic creation, updating, and resolution of incidents in Callgoose SQIBS based on events generated in Langsmith such as evaluation failures, dataset drift indicators, LLM execution anomalies, latency violations, or custom monitoring callbacks.
The guide includes steps for configuring Langsmith webhooks, setting up event callbacks, creating API filters in Callgoose SQIBS, and troubleshooting.
Prerequisites
- Langsmith account with permissions to configure Workspace Settings, Webhooks, and Evaluation settings.
- Callgoose SQIBS account with permissions to create API filters.
- Webhook/API endpoint generated in Callgoose SQIBS.
- API Token from Callgoose SQIBS.
1. Obtain API Token and Endpoint Details
To integrate with Callgoose SQIBS, first obtain an API token and endpoint URL.
1.1 Generate an API Token
Follow the document: How to Create API Token in Callgoose SQIBS.
Copy the generated token for later use.
1.2 Find the API Endpoint
Refer to Callgoose SQIBS API Endpoint Documentation.
Your endpoint will follow the format:
https://****.callgoose.com/v1/process?from=Langsmith&token=xxxx
Store this URL securely.
2. Configuring Langsmith to Send Alerts
Langsmith supports Webhook Integrations and Callback Handlers that forward monitoring events, evaluation results, and run-level failures to external systems such as Callgoose SQIBS.
2.1 Install and Set Up Langsmith (If Not Already Using)
- Log in to your Langsmith workspace.
- Ensure you have permissions to manage workspace-level settings and integrations.
2.2 Create a Webhook Destination
In Langsmith:
- Navigate to Workspace Settings → Integrations.
- Click Add Integration.
- Select Webhook.
- Configure the Webhook:
- Integration Name: Callgoose SQIBS
- URL: Your API Endpoint (from Step 1.2)
- Method: POST
- Headers: Optional (leave blank unless token is passed via header)
- Content Type: application/json
- Save the integration.
- Use the Test Webhook option to verify that Callgoose receives an event.
2.3 Configure Run/Trace/Event Callbacks
Langsmith allows defining callback handlers for:
- Evaluation runs
- Dataset validations
- Tracing events
- LLM-chain failures
- Custom application-level monitoring events
When creating or editing any of these:
- Navigate to the corresponding Evaluation, Monitor, or Callback Handler section.
- Select Add Webhook Notification.
- Choose the Callgoose SQIBS webhook created earlier.
2.4 Define Webhook Payload Format
Langsmith automatically sends structured JSON payloads for run failures, evaluations, and anomalies.
A recommended clean structure for compatibility with Callgoose is:
{
"event_type": "$event_type",
"run_id": "$run_id",
"project": "$project_name",
"status": "$status",
"severity": "$severity",
"message": "$message",
"timestamp": "$timestamp",
"metadata": $metadata
}
Fields depend on the specific Langsmith event (e.g., evaluation, run failure, callback error).
2.5 Test Webhook Delivery
In Langsmith Integrations:
- Locate the Callgoose SQIBS Webhook.
- Click Test.
- Confirm that Callgoose SQIBS receives the test payload under API Logs.
3. Example JSON Payloads From Langsmith
3.1 Triggered Alert Payload
{
"event_type": "run_error",
"run_id": "ls-run-98234",
"project": "Chatbot-Production",
"status": "triggered",
"severity": "CRITICAL",
"message": "Model response length exceeded allowed maximum.",
"timestamp": "2025-01-14T12:12:00Z",
"metadata": {
"model": "gpt-4o-mini",
"latency_ms": 5400,
"max_allowed_latency": 2000
}
}
3.2 Resolved Alert Payload
{
"event_type": "run_recovered",
"run_id": "ls-run-98234",
"project": "Chatbot-Production",
"status": "resolved",
"severity": "INFO",
"timestamp": "2025-01-14T12:25:00Z"
}
4. Configuring Callgoose SQIBS
4.1 Create API Filters
API filters are required to map incoming Langsmith payloads to incidents.
Use fields from the Langsmith JSON payload.
4.1.1 Trigger Filter (For Creating Incidents)
Configure the filter as follows:
- Payload JSON Key: "status"
- Key Value Contains: triggered
- Map Incident With: "run_id"
- Incident Title From: "message"
- Incident Description From: Leave empty to include full payload automatically.
4.1.2 Resolve Filter (For Resolving Incidents)
Configure:
- Payload JSON Key: "status"
- Key Value Contains: resolved
- Incident Mapped With: "run_id"
This ensures the correct incident resolves based on the matching run_id.
5. Testing and Validation
5.1 Triggering Alerts
In Langsmith:
- Modify an evaluation rule or callback to intentionally cause a failure (e.g., set max_latency threshold artificially low).
- Trigger a test run.
- Verify that:
- Callgoose API Logs show the incoming event.
- A new incident is created.
5.2 Resolving Alerts
In Langsmith:
- Revert the thresholds.
- Allow the next evaluation or run to pass successfully.
- Confirm that:
- A resolved event is sent.
- The incident in Callgoose auto-resolves.
6. Debugging and Troubleshooting
- No incident created
- Check if Langsmith sent the webhook.
- Confirm the correct "status" field in the payload.
- Check whether the Trigger Filter matches "triggered" exactly.
- 401 Unauthorized
- Verify the API Token in the URL.
- Empty or incorrect data in incident
- Check if Langsmith webhook payload includes the expected keys.
- Ensure the Payload Template is correctly set.
- Incident not resolving
- Ensure "run_id" matches in both trigger & resolve events.
- Confirm "status": "resolved" is present in resolved webhook.
- Payload schema mismatch
- Langsmith payloads differ per event type.
- Review Callgoose API Logs and adjust filters accordingly.
7. Security Considerations
- Keep your Callgoose API token private.
- Use HTTPS for all webhook calls.
- Rotate API tokens periodically.
- Use Langsmith’s Test button before saving integrations.
- Avoid exposing tokens in logs or frontend interfaces.
8. Conclusion
By following this guide, you can integrate Langsmith with Callgoose SQIBS for seamless incident creation and automated resolution for LLM evaluation failures, latency anomalies, model drifts, callback errors, and custom monitoring events.
This ensures your AI/LLM Ops teams get immediate visibility into production issues and can respond faster with automated workflows and real-time insights.
For further customization or advanced use cases, refer to the official documentation for both LangSmith and Callgoose SQIBS:
