Pagecrawl.io

Overview

This document provides a detailed guide to integrating Pagecrawl.io with Callgoose SQIBS for real-time Incident Management, Incident Auto Remediation, Event-Driven Automation, and other automation purposes. The integration enables automatic creation, updating, and resolution of incidents in Callgoose SQIBS based on alerts triggered in Pagecrawl.io. The guide includes steps for setting up alerts in Pagecrawl.io, configuring webhook notifications, creating API filters in Callgoose SQIBS, and troubleshooting.

Prerequisites

• Pagecrawl.io Account: Access to Pagecrawl.io for creating alerts and managing notifications.
• Callgoose SQIBS Account: With valid privileges to set up API filters and receive notifications.
• Webhook/API Endpoint: Available in Callgoose SQIBS to receive alerts from Pagecrawl.io.

1. Obtain API Token and Endpoint Details

To integrate with Callgoose SQIBS, you first need to obtain an API token and find the API endpoint details.

• Generate an API Token:
• Follow the guide on How to Create API Token in Callgoose SQIBS.
• Find the API Endpoint:
• Refer to the Callgoose SQIBS API Endpoint Documentation to get the endpoint details where the JSON payloads from Pagecrawl.io will be sent.

2. Debugging and Troubleshooting

You can enable debugging in the API tokens used with Pagecrawl.io notifications for troubleshooting purposes.

• Enable Debugging:
• You can update the debug value when adding or updating an API token.
• When API tracking is enabled, logs are stored in the API log section for your review. The debugging option will automatically disable after 48 hours.
• When API tracking is turned off, no logs are saved in the API log.
• Using API Log for Troubleshooting:
• The API log provides detailed information on all API calls made to Callgoose SQIBS.
• You can check the JSON values in each API log entry for troubleshooting purposes.
• Use the information in the API log to create or refine API filters to ensure incidents are created correctly based on the API payloads received.
• Callgoose SQIBS creates incidents according to your API filter configuration, giving you full control over how alerts from different services trigger incidents and alerts for your support team or automation processes.

3. Configuring Pagecrawl.io to Send JSON Payloads

Follow these steps to set up monitoring, alerts, and webhook integrations in Pagecrawl.io, ensuring that the JSON payloads generated match the required format for Callgoose SQIBS.

3.1 Configuring Your Track Page in Pagecrawler.io

• Log In
• Navigate to Pagecrawler.io and log in using your credentials.
• Create a New Tracked Page
• Click on “Track New Page” in the dashboard.
• Select Preference: For example, choose the Single URL option.
• Enter Details:
• URL Address: Type in the target URL.
• Title: Provide a title for the page.
• Configure Details & Tracked Elements:
• Define any specific page details and elements you want to monitor.
• Save: Click “Save” to create the tracked page.

3.2 Configuring the Webhook Integration

• Access Integrations
• After saving your track page, navigate to the “Integrations” section within Pagecrawler.io.
• Set Up a New Webhook
• Under the “Other” category, click on “Webhooks.”
• Click on “New Webhook” to start configuring the webhook.
• Paste the Endpoint URL
• Use the API Endpoint URL provided by callgoose sqibs.
• Remember: The final API endpoint might differ based on authentication and additional URL parameters.
• For more details on the endpoint structure and parameters, refer to the callgoose sqibs API documentation.
• Select Optional Page Notification
• If desired, choose the specific tracked page for which you want to receive notifications. This step is optional if you wish to limit webhook notifications to particular pages.
• Add the Webhook
• Click “Add” to save your webhook configuration.
• Testing the Integration
• Once the webhook is added, you can test the integration by clicking the “Test” button.
• Note: The test will only succeed if your tracked page has been executed (i.e., Pagecrawler.io has processed the page at least once).

3.3 Finalizing and Testing

• Validate the Integration:
• Trigger the alert condition manually if possible to verify that the correct JSON payload is sent to Callgoose SQIBS.

4. Configuring Callgoose SQIBS

4.1 Create API Filters in Callgoose SQIBS

To correctly map incidents from the Pagecrawl.io alerts, you need to create API filters based on the JSON payloads received.

4.1.1 Example JSON Payloads from Pagecrawler.io

Alert Triggered

json

{
  "id": 316507439,
  "title": "www.httpbin.org",
  "status": "ok",
  "content_type": "text/html; charset=utf-8",
  "visual_diff": null,
  "changed_at": "2025-02-10T09:38:16.000000Z",
  "contents": "This page isn’t working\n\nIf the problem continues, contact the site owner.\n\nHTTP ERROR 400\nReload",
  "elements": 1,
  "original": null,
  "difference": null,
  "human_difference": null,
  "page_screenshot_image": "https://pagecrawl.io/api/changes/12151960/checks/316507439/screenshot?api_token=****",
  "text_difference_image": "https://pagecrawl.io/api/changes/12151960/checks/316507439/diff.png?api_token=****",
  "html_difference": "<h1 style=\"margin-bottom: 6px\">Element</h1>\n<ins class=\"diffins\"><span style=\"color: #52c41a;background-color: #eaffea;font-weight: 600\">This page isn’t working<br />\n<br />\nIf the problem continues, contact the site owner.<br />\n<br />\nHTTP ERROR 400<br />\nReload</span></ins>",
  "markdown_difference": "# Element\n*This page isn’t working\n\nIf the problem continues, contact the site owner.\n\nHTTP ERROR 400\nReload*\n",
  "page": {
    "id": 12151960,
    "name": "www.httpbin.org",
    "url": "https://www.httpbin.org",
    "url_tld": "www.httpbin.org",
    "slug": "avwqm1aheibgon2x",
    "link": "https://pagecrawl.io/app/pages/****",
    "folder": null
  },
  "page_elements": [
    {
      "id": 100777251,
      "label": "Element",
      "type": "fullpage",
      "contents": "This page isn’t working\n\nIf the problem continues, contact the site owner.\n\nHTTP ERROR 400\nReload",
      "elements": 1,
      "original": null,
      "difference": null,
      "human_difference": null,
      "hash": "5f09e59b5ed2795ef81808574b80c241",
      "changed": 0
    }
  ],
  "previous_check": null
}

4.2 Configuring API Filters

4.2.1 Integration Templates

If you see an Pagecrawl.io integration template in the "Select Integration Template" dropdown in the API filter settings, you can use it to automatically add the necessary Trigger and Resolve filters along with other values. The values added by the template can be modified to customize the integration according to your requirements.

4.2.2 Manually Add/Edit the Filter

• Trigger Filter (For Creating Incidents):
• Payload JSON Key: "status"
• Key Value Contains: null
• Map Incident With: null
• Incident Title From: null
• Incident Description From: Leave this empty unless you want to use a specific key-value from the JSON payload. If a key is entered, only the value for that key will be used as the Incident Description instead of the full JSON. By default, the Incident Description will include the full JSON values.
• Example: If you use the "contents" key in the Incident Description From field, the incident description will be the value of the "contents" key. In the example JSON payload provided earlier, this would result in a description like "This page isn’t working\n\nIf the problem continues, contact the site owner.\n\nHTTP ERROR 400\nReload".
• Resolve Filter (For Resolving Incidents):
• Payload JSON Key: null
• Key Value Contains: null
• Incident Mapped With: null

Refer to the API Filter Instructions and FAQ for more details.

4.3 Finalizing Setup

• Save the API Filters:
• Ensure that the filters are correctly configured and saved in Callgoose SQIBS.
• Double-check that all key mappings, incident titles, and descriptions are correctly aligned with the payload structure sent by Pagecrawl.io.

5. Testing and Validation

5.1 Triggering Alerts

• Simulate a Monitoring Alert:
• Trigger a condition in Pagecrawl.io that causes an alert (e.g., when the server or website is down).
• Verify that an incident is created in Callgoose SQIBS with the correct information.

6. Security Considerations

• API Security: Ensure that the Callgoose SQIBS API endpoint is correctly configured and that the API token is securely stored and used.
• Pagecrawl.io Permissions: Confirm that the webhook in Pagecrawl.io has appropriate permissions to send alerts and data to Callgoose SQIBS.

7. Troubleshooting

• No Incident Created: If no incident is created, verify that the webhook URL in Pagecrawl.io is correct and that the JSON payload structure matches the API filters configured in Callgoose SQIBS.
• Incident Not Resolved: Ensure that the resolve filter in Callgoose SQIBS is correctly configured and that the JSON payload sent by Pagecrawl.io matches the expected structure.

8. Conclusion

This guide provides a comprehensive overview of how to integrate Pagecrawl.io with Callgoose SQIBS for effective incident management. By following the steps outlined, you can ensure that alerts from Pagecrawl.io are automatically reflected as incidents in Callgoose SQIBS, with proper resolution tracking when the issues are resolved.

For further customization or advanced use cases, refer to the official documentation for both Pagecrawl.io and Callgoose SQIBS:

• Pagecrawl.io Documentation
• Callgoose SQIBS API Token Documentation
• Callgoose SQIBS API Endpoint Documentation
• API Filter Instructions and FAQ
• How to Send API

This documentation will guide you through the integration process, ensuring that your incidents are managed effectively within Callgoose SQIBS based on real-time alerts from Pagecrawl.io.