Falco

Integrating Falco with Callgoose SQIBS API

Overview

This document provides a detailed guide to integrating Falco with Callgoose SQIBS for real-time Incident Management, Incident Auto Remediation, Event-Driven Automation, and other automation purposes. The integration enables automatic creation, updating, and resolution of incidents in Callgoose SQIBS based on alerts triggered in Falco. The guide includes steps for setting up alerts in Falco, configuring webhook notifications, creating API filters in Callgoose SQIBS, and troubleshooting.

This guide focuses on a Kubernetes (Minikube) environment where alerts are routed through Prometheus Alertmanager. This is a robust production setup that allows for advanced alert grouping, silencing, and routing.

The alert pipeline we will configure is: Falco -> Falcosidekick -> Alertmanager -> Callgoose SQIBS

Prerequisites

• Falco Account: Access to Falco for creating alerts and managing notifications.
• Callgoose SQIBS Account: With valid privileges to set up API filters and receive notifications.
• Webhook/API Endpoint: Available in Callgoose SQIBS to receive alerts from Falco.

1. Obtain API Token and Endpoint Details

To integrate with Callgoose SQIBS, you first need to obtain an API token and find the API endpoint details.

• Generate an API Token: Follow the guide on How to Create API Token in Callgoose SQIBS.
• Find the API Endpoint: Refer to the Callgoose SQIBS API Endpoint Documentation to get the endpoint details where the JSON payloads from Falco will be sent.

2. Installing Falco

First, you need to install Falco. The method depends on your environment.

2.1. For Kubernetes (Minikube) - Primary Method

The standard method for installing Falco on Kubernetes is by using the official Helm chart. This will deploy Falco as a DaemonSet to monitor all your nodes.

Shell:

# 1. Add the Falco Helm repository
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# 2. Install Falco + Falcosidekick
# We use --create-namespace to make the 'falco' namespace
# We add --set tty=true as recommended by the quickstart
# This install enables Falcosidekick, which is required to forward
# alerts to Alertmanager.
helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set tty=true \
  --set falcosidekick.enabled=true

2.2. For Linux Hosts (Debian/Ubuntu)

While this guide's configuration focuses on Kubernetes, you can install Falco on a standalone Linux host as follows. These steps are from the official Falco documentation.

Shell:

# 1. Add the Falco GPG key
curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \
sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg

# 2. Add the Falco repository
sudo bash -c 'cat << EOF > /etc/apt/sources.list.d/falcosecurity.list
deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main
EOF'

# 3. Update package lists
sudo apt-get update -y

# 4. Install dialog (used for the installation wizard)
sudo apt-get install -y dialog

# 5. Install Falco
# You will be prompted to select a driver (e.g., kmod, ebpf).
sudo apt-get install -y falco

2.3. For Linux Hosts (CentOS/RHEL/Fedora)

These steps are from the official Falco documentation for package installs.

Shell:

# 1. Add the Falco GPG key
sudo rpm --import https://falco.org/repo/falcosecurity-packages.asc

# 2. Add the Falco repository
sudo curl -s -o /etc/yum.repos.d/falcosecurity.repo https://falco.org/repo/falcosecurity-rpm.repo

# 3. Install kernel headers and make (dependencies)
sudo yum install -y kernel-devel-$(uname -r) make

# 4. Install Falco
sudo yum install -y falco

3. Configuring the Alertmanager Pipeline

This section details how to configure the alert forwarding from Falco to Callgoose SQIBS using Alertmanager.

3.1 Configure Falco + Falcosidekick to Talk to Alertmanager

When you installed the Helm chart (Step 2.1), you enabled falcosidekick. Now, you must configure Falcosidekick to send its alerts to your Alertmanager service. You can do this during the Helm install or with a helm upgrade.

Shell:

# Replace <your-alertmanager-service-url> with the DNS name of your
# Alertmanager service (e.g., "http://alertmanager.alertmanager.svc.cluster.local:9093")

helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set falcosidekick.enabled=true \
  --set falcosidekick.config.alertmanager.hostport="<your-alertmanager-service-url>"

This command tells Falcosidekick to forward all incoming Falco alerts to your Alertmanager instance.

3.2 Configure Alertmanager to Talk to Callgoose SQIBS

You need to edit the Alertmanager configuration (alertmanager.yml) to add Callgoose SQIBS as a "receiver". In Kubernetes, this is stored in a ConfigMap.

Run this command to edit the Alertmanager ConfigMap:

kubectl -n alertmanager edit configmap alertmanager-config

Example Yaml:

YAML

apiVersion: v1
data:
  alertmanager.yaml: |
    global:
      resolve_timeout: 5m

    route:
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 30s
      repeat_interval: 1m
      receiver: 'callgoose-sqibs'

    receivers:
      - name: 'callgoose-sqibs'
        webhook_configs:
          - url: 'https://****.callgoose.com/v1/process?from=FromValue&token=xxxx'
            send_resolved: true

    inhibit_rules:
      - source_match:
          severity: 'critical'
        target_match:
          severity: 'warning'
        equal: ['alertname', 'dev', 'instance']
kind: ConfigMap
metadata:
  creationTimestamp: "2025-10-17T12:55:37Z"
  name: alertmanager-config
  namespace: alertmanager
  resourceVersion: "447378"
  uid: 2ad84fd4-39d0-43ad-988b-51ab7583fb27

After saving and closing the editor, restart Alertmanager to apply the new configuration:

kubectl -n alertmanager rollout restart deployment alertmanager

4. Configuring Callgoose SQIBS

4.1 Create API Filters in Callgoose SQIBS

To correctly map incidents from the Aha alerts, you need to create API filters based on the JSON payloads received.

4.1.1 Example JSON Payloads from Aha

Alert Triggered

JSON

{
	"receiver": "callgoose-sqibs",
	"status": "firing",
	"alerts": [
		{
			"status": "firing",
			"labels": {
				"container_id": "****",
				"container_image_repository": "openjdk",
				"container_image_tag": "17-jdk",
				"container_name": "****",
				"eventsource": "syscall",
				"evt_time": "****",
				"evt_type": "openat",
				"fd_name": "/etc/****",
				"hostname": "****",
				"k8s_ns_name": "default",
				"k8s_pod_name": "****",
				"priority": "Warning",
				"proc_aname_2": "\u003cNA\u003e",
				"proc_cmdline": "****",
				"proc_exepath": "****",
				"proc_name": "cat",
				"proc_pname": "containerd-shim",
				"proc_tty": "****",
				"rule": "Read sensitive file untrusted",
				"severity": "warning",
				"source": "falco",
				"tags": "****",
				"user_loginuid": "****",
				"user_name": "root",
				"user_uid": "****"
			},
			"annotations": {
				"description": "04:11:22.414263623: Warning Sensitive file opened for reading by non-trusted program | file=/etc/shadow gparent=\u003cNA\u003e ggparent=\u003cNA\u003e gggparent=\u003cNA\u003e evt_type=openat user=root user_uid=0 user_loginuid=7014 process=cat proc_exepath=/usr/bin/coreutils parent=containerd-shim command=cat --coreutils-prog-shebang=cat /usr/bin/cat /etc/shadow terminal=34816 container_id=b9ca5122431b container_name=k8s_builder_javac-pod_default_2e807c03-3dfc-4da3-8a43-f5bcbf87258c_78 container_image_repository=openjdk container_image_tag=17-jdk k8s_pod_name=javac-pod k8s_ns_name=default",
				"info": "****",
				"summary": "Read sensitive file untrusted"
			},
			"startsAt": "****",
			"endsAt": "0001-01-01T00:00:00Z",
			"generatorURL": "",
			"fingerprint": "****"
		}
	],
	"groupLabels": {},
	"commonLabels": {
		"container_id": "****",
		"container_image_repository": "openjdk",
		"container_image_tag": "17-jdk",
		"container_name": "****",
		"eventsource": "syscall",
		"evt_time": "****",
		"evt_type": "openat",
		"fd_name": "/etc/****",
		"hostname": "****",
		"k8s_ns_name": "default",
		"k8s_pod_name": "****",
		"priority": "Warning",
		"proc_aname_2": "\u003cNA\u003e",
		"proc_cmdline": "****",
		"proc_exepath": "****",
		"proc_name": "cat",
		"proc_pname": "containerd-shim",
		"proc_tty": "****",
		"rule": "Read sensitive file untrusted",
		"severity": "warning",
		"source": "falco",
		"tags": "****",
		"user_loginuid": "****",
		"user_name": "root",
		"user_uid": "****"
	},
	"commonAnnotations": {
		"description": "04:11:22.414263623: Warning Sensitive file opened for reading by non-trusted program | file=/etc/shadow gparent=\u003cNA\u003e ggparent=\u003cNA\u003e gggparent=\u003cNA\u003e evt_type=openat user=root user_uid=0 user_loginuid=7014 process=cat proc_exepath=/usr/bin/coreutils parent=containerd-shim command=cat --coreutils-prog-shebang=cat /usr/bin/cat /etc/shadow terminal=34816 container_id=b9ca5122431b container_name=k8s_builder_javac-pod_default_2e807c03-3dfc-4da3-8a43-f5bcbf87258c_78 container_image_repository=openjdk container_image_tag=17-jdk k8s_pod_name=javac-pod k8s_ns_name=default",
		"info": "****",
		"summary": "Read sensitive file untrusted"
	},
	"externalURL": "****",
	"version": "4",
	"groupKey": "{}:{}",
	"truncatedAlerts": 0
}

Alert Resolved

JSON

{
  "receiver": "callgoose-sqibs",
  "status": "resolved",
  "alerts": [
    {
      "status": "resolved",
      "labels": {
        "container_id": "****",
        "container_image_repository": "openjdk",
        "container_image_tag": "17-jdk",
        "container_name": "****",
        "eventsource": "syscall",
        "evt_time": "****",
        "evt_type": "openat",
        "fd_name": "/etc/****",
        "hostname": "****",
        "k8s_ns_name": "default",
        "k8s_pod_name": "****",
        "priority": "Warning",
        "proc_aname_2": "<NA>",
        "proc_cmdline": "****",
        "proc_exepath": "****",
        "proc_name": "cat",
        "proc_pname": "containerd-shim",
        "proc_tty": "****",
        "rule": "Read sensitive file untrusted",
        "severity": "warning",
        "source": "falco",
        "tags": "****",
        "user_loginuid": "****",
        "user_name": "root",
        "user_uid": "****"
      },
      "annotations": {
        "description": "04:11:22.414263623: Warning Sensitive file opened for reading by non-trusted program | file=/etc/shadow gparent=<NA> ggparent=<NA> gggparent=<NA> evt_type=openat user=root user_uid=0 user_loginuid=7014 process=cat proc_exepath=/usr/bin/coreutils parent=containerd-shim command=cat --coreutils-prog-shebang=cat /usr/bin/cat /etc/shadow terminal=34816 container_id=b9ca5122431b container_name=k8s_builder_javac-pod_default_2e807c03-3dfc-4da3-8a43-f5bcbf87258c_78 container_image_repository=openjdk container_image_tag=17-jdk k8s_pod_name=javac-pod k8s_ns_name=default",
        "info": "****",
        "summary": "Read sensitive file untrusted"
      },
      "startsAt": "****",
      "endsAt": "****",
      "generatorURL": "",
      "fingerprint": "****"
    }
  ],
  "groupLabels": {},
  "commonLabels": {
    "container_id": "****",
    "container_image_repository": "openjdk",
    "container_image_tag": "17-jdk",
    "container_name": "****",
    "Methods": "syscall",
    "evt_time": "****",
    "evt_type": "openat",
    "fd_name": "/etc/****",
    "hostname": "****",
    "k8s_ns_name": "default",
    "k8s_pod_name": "****",
    "priority": "Warning",
    "proc_aname_2": "<NA>",
    "proc_cmdline": "****",
    "proc_exepath": "****",
    "proc_name": "cat",
    "proc_pname": "containerd-shim",
    "proc_tty": "****",
    "rule": "Read sensitive file untrusted",
    "severity": "warning",
    "source": "falco",
    "tags": "****",
    "user_loginuid": "****",
    "user_name": "root",
    "user_uid": "****"
  },
  "commonAnnotations": {
    "description": "04:11:22.414263623: Warning Sensitive file opened for reading by non-trusted program | file=/etc/shadow gparent=<NA> ggparent=<NA> gggparent=<NA> evt_type=openat user=root user_uid=0 user_loginuid=7014 process=cat proc_exepath=/usr/bin/coreutils parent=containerd-shim command=cat --coreutils-prog-shebang=cat /usr/bin/cat /etc/shadow terminal=34816 container_id=b9ca5122431b container_name=k8s_builder_javac-pod_default_2e807c03-3dfc-4da3-8a43-f5bcbf87258c_78 container_image_repository=openjdk container_image_tag=17-jdk k8s_pod_name=javac-pod k8s_ns_name=default",
    "info": "****",
    "summary": "Read sensitive file untrusted"
  },
  "externalURL": "****",
  "version": "4",
  "groupKey": "{}:{}",
  "truncatedAlerts": 0
}

4.2 Configuring API Filters

4.2.1 Integration Templates

If you see a Falco integration template in the "Select Integration Template" dropdown in the API filter settings, you can use it to automatically add the necessary Trigger and Resolve filters along with other values. The values added by the template can be modified to customize the integration according to your requirements.

4.2.2 Manually Add/Edit the Filter

Trigger Filter (For Creating Incidents):

• Payload JSON Key: "status"
• Key Value Contains: [firing]
• Map Incident With: "groupKey"
• Incident Title From: null
• Incident Description From: Leave this empty unless you want to use a specific key-value from the JSON payload. If a key is entered, only the value for that key will be used as the Incident Description instead of the full JSON. By default, the Incident Description will include the full JSON values.
• Example: If you use the "alerts".[0]."annotations"."description" key in the Incident Description From field, the incident description will be the value of the "alerts".[0]."annotations"."description" key. In the example JSON payload provided earlier, this would result in a description like "04:11:22.414263623: Warning Sensitive file opened for reading by non-trusted program | file=/etc/shadow gparent=<NA> ggparent=<NA> gggparent=<NA> evt_type=openat user=root user_uid=0 user_loginuid=7014 process=cat proc_exepath=/usr/bin/coreutils parent=containerd-shim command=cat --coreutils-prog-shebang=cat /usr/bin/cat /etc/shadow terminal=34816 container_id=b9ca5122431b container_name=k8s_builder_javac-pod_default_2e807c03-3dfc-4da3-8a43-f5bcbf87258c_78 container_image_repository=openjdk container_image_tag=17-jdk k8s_pod_name=javac-pod k8s_ns_name=default".

Resolve Filter (For Resolving Incidents):

• Payload JSON Key: "status"
• Key Value Contains: [resolved]
• Incident Mapped With: "groupKey"

Refer to the API Filter Instructions and FAQ for more details.

4.3 Finalizing Setup

• Save the API Filters: Ensure that the filters are correctly configured and saved in Callgoose SQIBS.
• Double check that all key mappings, incident titles, and descriptions are correctly aligned with the payload structure sent by Falco.

5. Verifying the Integration

You can now verify that the integration is working.

• Trigger a Test Alert: Exec into any pod in your Kubernetes cluster and run a command that triggers a default Falco rule. Example:
• Shell:

# Find a pod to exec into
POD_NAME=$(kubectl get pods -n default -o jsonpath='{.items[0].metadata.name}')

# Trigger an alert
kubectl exec -it $POD_NAME -n default -- cat /etc/shadow

• Check Your Callgoose SQIBS Dashboard: Log in to your Callgoose SQIBS platform. You should see a new alert corresponding to the event you triggered.

6. Adding a Custom Falco Rule

One of the most powerful features of Falco is adding your own rules. In Kubernetes, the easiest way to do this is by providing a custom rules file to the Helm chart.

6.1 Create a Custom Rules YAML File

Create a file locally named falco-custom-rules.yaml.

Here is an example rule that will trigger an alert (Priority: WARNING) any time a file is created in the /bin directory, which is suspicious behavior. This is the rule triggered by the touch /bin/test-alert command in the verification step.

falco-custom-rules.yaml

YAML:

- rule: "Create file in /bin"desc: "Detects a file being created in the /bin directory"condition: "evt.type = creat and evt.dir = > and fd.directory = /bin"output: "File created in /bin (user=%user.name command=%proc.cmdline file=%fd.name)"priority: WARNINGtags: [filesystem, mitre_persistence]

6.2 Load the Custom Rule into Falco

You can now use a helm upgrade command to pass this file to your Falco installation. This command loads your custom rules file without disabling the default rules.

Shell:

# Run this from the same directory where you saved falco-custom-rules.yaml

helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --reuse-values \
  -f falco-custom-rules.yaml

• --reuse-values: This is important. It keeps all your previous settings (like the Alertmanager URL).
• -f falco-custom-rules.yaml: This tells Helm to apply the rules from your new file.

Falco will automatically restart, and your new rule will be active. You can test it by running the touch /bin/test-alert command again and watching the logs.

7. Debugging and Troubleshooting

You can enable debugging in the API tokens used with Falco notifications for troubleshooting purposes.

• Enable Debugging: You can update the debug value when adding or updating an API token. When API tracking is enabled, logs are stored in the API log section for your review. The debugging option will automatically disable after 48 hours. When API tracking is turned off, no logs are saved in the API log.
• Using API Log for Troubleshooting: The API log provides detailed information on all API calls made to Callgoose SQIBS. You can check the JSON values in each API log entry for troubleshooting purposes. Use the information in the API log to create or refine API filters to ensure incidents are created correctly based on the API payloads received.

Callgoose SQIBS creates incidents according to your API filter configuration, giving you full control over how alerts from different services trigger incidents and alerts for your support team or automation processes.

Reference - Falco