Argo CD

Overview

This document provides a detailed guide to integrating Argo CD with Callgoose SQIBS for real-time Incident Management, Incident Auto Remediation, Event-Driven Automation, and other automation purposes. The integration enables automatic creation, updating, and resolution of incidents in Callgoose SQIBS based on alerts triggered in Argo CD. The guide includes steps for setting up alerts in Argo CD, configuring webhook notifications, creating API filters in Callgoose SQIBS, and troubleshooting.

Argo CD notifications will be delivered to Callgoose SQIBS through the following pipeline:

Argo CD -> Argo CD Notifications -> Callgoose SQIBS (Webhook Endpoint)

Prerequisites

• Kubernetes cluster (minikube, kind, EKS, GKE, etc.)
• kubectl configured
• helm (optional for some install steps)
• Argo CD admin access
• Callgoose SQIBS API token & endpoint

1. Local / Quick Argo CD installation (single-cluster, dev -> production checklist)

1.1 Install Argo CD (official manifests)

This installs Argo CD server, repo-server, application-controller, redis, etc.

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Wait for pods:

kubectl -n argocd rollout status deploy/argocd-server
kubectl -n argocd get pods

Get admin password (initial):

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Port-forward (local access;):
kubectl -n argocd port-forward svc/argocd-server 8080:443
# then open https://localhost:8080

• For production: use an Ingress or LoadBalancer, enable TLS, configure RBAC, enable SSO (OIDC), and secure the API server.

1.2 Install argocd-notifications (controller + config) — two options

1.2.1 Option A — Helm (recommended for prod)

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd-notifications argo/argo-cd-notifications \
  --namespace argocd \
  --set fullnameOverride=argocd-notifications

1.2.2 Option B — Manifests (quick)

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-notifications/stable/manifests/install.yaml

Verify:

kubectl -n argocd get deploy argocd-notifications-controller
kubectl -n argocd logs deploy/argocd-notifications-controller

2. Production-ready argocd-notifications ConfigMap

• Below is a full example argocd-notifications-cm you can use. It includes:
• Webhook service for Callgoose SQIBS (with optional HMAC header)
• Several templates (sync-failed, degraded, out-of-sync, healthy/resolve)
• Triggers for production-level usage (rate limiting and grouping)
• Example of using labels and annotations to enrich payloads
• Replace https://YOUR_CALLGOOSE_ENDPOINT and CALLGOOSE_TOKEN before applying. For more details on the Callgoose SQIBS API Endpoint, refer to the documentation: Callgoose SQIBS API Endpoint.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.webhook.callgoose-sqibs: |
    url: https://YOUR_CALLGOOSE_ENDPOINT/sqibs-api/v1/process?from=ArgoCD&token=CALLGOOSE_TOKEN
    method: POST
    headers:
      Content-Type: application/json
      X-Callgoose-Signature: "sha256={{.secrets.callgoose_secret | hmacsha256 .body}}"

  template.app-sync-failed: |
    title: ArgoCD - Sync Failed - {{.app.metadata.name}}
    body: |
      {
        "event": "sync_failed",
        "app": "{{.app.metadata.name}}",
        "project": "{{.app.spec.project}}",
        "namespace": "{{.app.spec.destination.namespace}}",
        "server": "{{.app.spec.destination.server}}",
        "sync_status": "{{.app.status.sync.status}}",
        "health_status": "{{.app.status.health.status}}",
        "message": "{{.app.status.operationState.message}}",
        "revision": "{{.app.status.sync.revision}}",
        "operation": "{{.app.status.operationState.operation}}",
        "started_at": "{{.app.status.operationState.startedAt}}",
        "finished_at": "{{.app.status.operationState.finishedAt}}",
        "links": {
          "ui": "{{.appstatus.uiURL}}"
        }
      }

  template.app-degraded: |
    title: ArgoCD - Degraded - {{.app.metadata.name}}
    body: |
      {
        "event": "degraded",
        "app": "{{.app.metadata.name}}",
        "project": "{{.app.spec.project}}",
        "namespace": "{{.app.spec.destination.namespace}}",
        "health_status": "{{.app.status.health.status}}",
        "message": "{{.app.status.operationState.message}}",
        "time": "{{.app.status.operationState.finishedAt}}"
      }

  template.app-out-of-sync: |
    title: ArgoCD - OutOfSync - {{.app.metadata.name}}
    body: |
      {
        "event": "out_of_sync",
        "app": "{{.app.metadata.name}}",
        "desired": "{{.app.status.sync.comparedTo}}",
        "status": "{{.app.status.sync.status}}",
        "time": "{{.app.status.operationState.finishedAt}}"
      }

  template.app-healthy: |
    title: ArgoCD - Resolved/Healthy - {{.app.metadata.name}}
    body: |
      {
        "event": "healthy",
        "app": "{{.app.metadata.name}}",
        "project": "{{.app.spec.project}}",
        "namespace": "{{.app.spec.destination.namespace}}",
        "health_status": "{{.app.status.health.status}}",
        "message": "Application is healthy and synced",
        "time": "{{.app.status.operationState.finishedAt}}"
      }

  trigger.on-sync-failed: |
    - description: Send on sync failure (failed operation)
      condition: app.status.operationState.phase == 'Failed'
      group: app.metadata.name
      send: [app-sync-failed]

  trigger.on-degraded: |
    - description: App health is Degraded
      condition: app.status.health.status == 'Degraded'
      group: app.metadata.name
      send: [app-degraded]

  trigger.on-out-of-sync: |
    - description: App is OutOfSync for longer than X minutes
      condition: app.status.sync.status == 'OutOfSync' && app.status.operationState.finishedAt != ''
      group: app.metadata.name
      send: [app-out-of-sync]

  trigger.on-healthy: |
    - description: Application returned to healthy state (resolve)
      condition: app.status.health.status == 'Healthy' && app.status.operationState.phase == 'Succeeded'
      group: app.metadata.name
      send: [app-healthy]

  subscriptions: |
    - recipients:
      - callgoose-sqibs
      triggers:
      - on-sync-failed
      - on-degraded
      - on-out-of-sync
      - on-healthy

How to apply:

kubectl apply -f argocd-notifications-cm.yaml -n argocd
kubectl rollout restart deployment argocd-notifications-controller -n argocd

3. Example JSON payloads

3.1 Sync failed — Triggered payload

{
  "event": "sync_failed",
  "app": "frontend",
  "project": "default",
  "namespace": "production",
  "server": "https://kubernetes.default.svc",
  "sync_status": "OutOfSync",
  "health_status": "Degraded",
  "message": "Sync operation failed: Failed to apply manifests: image pull backoff",
  "revision": "abcd1234",
  "operation": "Sync",
  "started_at": "2025-11-14T08:40:00Z",
  "finished_at": "2025-11-14T08:41:02Z",
  "links": {
    "ui": "https://argocd.company.example/applications/frontend"
  }
}

3.2 Healthy/resolved — Resolved payload

{
  "event": "healthy",
  "app": "frontend",
  "project": "default",
  "namespace": "production",
  "health_status": "Healthy",
  "message": "Application is healthy and synced",
  "time": "2025-11-14T08:45:10Z"
}

4. Callgoose SQIBS: API Filter examples (update if templates changed)

Trigger Filter — Create incident on sync_failed / degraded / out_of_sync

• Payload JSON Key: event
• Key Value Contains: sync_failed OR degraded OR out_of_sync
• Map Incident With: app
• Incident Title: message
• Incident Description: leave empty to include full JSON OR use message to get "Sync operation failed: Failed to apply manifests: image pull backoff."

Resolve Filter — Resolve incident when app becomes healthy

• Payload JSON Key: event
• Key Value Contains: healthy
• Incident Mapped With: app

Refer to the API Filter Instructions and FAQ for more details.

5. Testing & verification

• Apply the ConfigMap and restart the notifications controller.
• Trigger a failing sync (deploy an invalid image or corrupt manifest) in Argo CD UI or by kubectl apply.
• Check argocd-notifications-controller logs for outgoing webhook delivery.
• kubectl -n argocd logs deploy/argocd-notifications-controller
• Verify the event in Callgoose SQIBS (incident created).
• Fix the issue. When Argo CD marks the app Healthy, verify the incident resolves.

6. Troubleshooting & debugging

• If no event arrives: check argocd-notifications-controller logs and whether the webhook URL is reachable.
• If wrong mapping: inspect raw JSON in Callgoose SQIBS API logs (enable debug on API token).
• If too many alerts (alert storm): add throttling/aggregation in triggers (group by app and add a minimal interval before resend) or configure controller retries in helm values.

7. Security recommendations

• Use HTTPS endpoints and prefer bearer auth headers. Store secrets in argocd-notifications-secret.
• Restrict network egress from the cluster to only the Callgoose endpoints.

8. Conclusion

This guide gives a production-ready recipe for integrating Argo CD with Callgoose SQIBS using argocd-notifications. Use the provided ConfigMap and templates as a starting point, update the Callgoose API filters to match the JSON keys, and test end-to-end using the example payloads. For advanced use cases (multi-cluster, per-namespace routing, or custom severity mapping), expand triggers and mapping keys accordingly.