PowerShell - Incident Auto Remediation

PowerShell - Incident Auto Remediation

This guide provides a step-by-step process to set up a PowerShell SSH connection to a Windows Server 2022 and utilize Callgoose SQIBS Incident Auto Remediation to automatically resolve alerts generated by any new monitoring or observability system.

1. Installing PowerShell 7.x on Windows Server 2022

PowerShell 7.x offers many improvements and new features that are essential for modern automation scenarios, such as incident remediation. Below are two ways to install PowerShell 7.x on Windows Server 2022.

Step 1: Download and Install PowerShell 7.x Manually

1. Visit the PowerShell GitHub releases page and download the PowerShell 7.x MSI for Windows.
2. Run the installer and follow the on-screen instructions.
3. After installation, verify the installation by running the following command:

powershell

pwsh --version

Step 2: Install PowerShell 7.x Using Windows PowerShell

1. Open Windows PowerShell and run the following command to download and install PowerShell 7.x:

powershell

iex "& { $(irm https://aka.ms/install-powershell.ps1) } -UseMSI"

1. After the installation, you can launch PowerShell 7.x by typing:

powershell

pwsh

2. Installing and Enabling SSH on Windows Server 2022

To enable SSH on Windows Server 2022, follow these steps:

Step 1: Install OpenSSH Server

1. Open PowerShell as an Administrator and install OpenSSH Server:

powershell

Get-WindowsCapability -Online | Where-Object Name -like 'OpenSSH*'
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0

1. (Replace OpenSSH.Server~~~~0.0.1.0 with the latest version if available.)
2. Start and configure the SSH service to start automatically:

powershell

Start-Service sshd
Set-Service -Name sshd -StartupType 'Automatic'

1. Confirm the SSH server is running:

powershell

Get-Service -Name sshd

1. Verify the SSH installation:

powershell

Get-WindowsCapability -Online | Where-Object Name -like 'OpenSSH*'

1. You should see OpenSSH.Server listed as installed.

Step 2: Configure the Firewall for SSH

Allow port 22 (default SSH port) through the Windows firewall to enable SSH access:

powershell

New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22

Step 3: Configure Trusted Hosts

If the Windows Server and Linux server are not part of the same domain, configure TrustedHosts to allow SSH connections:

1. View the current trusted hosts:

powershell

Get-Item WSMan:\localhost\Client\TrustedHosts

1. Add a trusted host:

powershell

Set-Item WSMan:\localhost\Client\TrustedHosts -Value "192.168.1.100"  # Replace with your Linux server IP

1. To append more trusted hosts:

powershell

$current = Get-Item WSMan:\localhost\Client\TrustedHosts
Set-Item WSMan:\localhost\Client\TrustedHosts -Value "$($current.Value),192.168.1.101"

Step 4: Test SSH Connection

On your Rocky Linux server, try connecting to the Windows Server via SSH:

bash

ssh Administrator@<Windows_Server_IP>

You should be prompted for the Administrator user’s password. Upon success, you'll be connected to the Windows Server.

3. Setting PowerShell 7.x as the Default for SSH

By default, when connecting via SSH, Windows opens a cmd.exe shell. Follow these steps to change the default shell to PowerShell 7.x:

1. Set PowerShell 7.x as the default shell for SSH:

powershell

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Program Files\PowerShell\7\pwsh.exe" -PropertyType String -Force

1. Restart the SSH service:

powershell

Restart-Service sshd

Now, SSH sessions will automatically open PowerShell 7.x.

4. Enabling SSH Logging (Optional)

To enable verbose logging for SSH:

1. Open the sshd_config file (usually located at C:\ProgramData\ssh\sshd_config).
2. Add or modify the following lines for logging:

plaintext

SyslogFacility LOCAL0
LogLevel VERBOSE

1. Restart the SSH service:

powershell

Restart-Service sshd

Logs will be generated in the C:\ProgramData\ssh\logs directory.

5. Creating Passwordless SSH Connections Between Linux and Windows

Passwordless connections are crucial for automating tasks like incident remediation. Follow these steps to configure passwordless SSH:

Step 1: Generate SSH Keys on Linux

1. On the Linux server, generate SSH keys:

bash

ssh-keygen -t ed25519

1. Copy the public key to the Windows Server:

bash

scp ~/.ssh/id_ed25519.pub Administrator@192.168.1.100:'C:\ProgramData\ssh\administrators_authorized_keys'

Step 2: Set Correct ACLs for Authorized Keys

1. On the Windows Server, set the correct ACLs for the authorized keys:

powershell

icacls.exe "C:\ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant "Administrators:F" /grant "SYSTEM:F"

1. Restart the SSH service:

powershell

Restart-Service sshd

6. Testing SSH and Executing PowerShell Scripts Remotely

Step 1: Test the SSH Connection

On the Linux server, run:

bash

ssh Administrator@192.168.1.100

If successful, this will log you into PowerShell 7.x on the Windows Server.

Step 2: Execute PowerShell Scripts via SSH

To execute PowerShell scripts remotely:

bash

ssh Administrator@192.168.1.100 "pwsh -File C:/path/to/script.ps1"

Example:

bash

ssh Administrator@192.168.1.100 "pwsh -File C:\Users\Administrator\check_disk_space.ps1"

Expected output:

plaintext

DeviceID VolumeName Size (GB) FreeSpace (GB)
-------- ---------- --------- --------------
C:                  199.39    186.29

Sample check_disk_space.ps1 Script

powershell

# This script checks disk space on all logical drives

$diskInfo = Get-CimInstance -ClassName Win32_LogicalDisk | 
            Where-Object { $_.DriveType -eq 3 } | 
            Select-Object DeviceID, VolumeName, 
                @{Name="Size (GB)"; Expression = { "{0:N2}" -f ($_.Size / 1GB) }}, 
                @{Name="FreeSpace (GB)"; Expression = { "{0:N2}" -f ($_.FreeSpace / 1GB) }}

# Output the disk space details
$diskInfo

7. Incident Auto Remediation Using PowerShell and Callgoose SQIBS

After setting up SSH and PowerShell communication between Linux and Windows, you can integrate Callgoose SQIBS to automatically trigger incident auto-remediation scripts.

Example Workflow

1. Monitoring System Alert: A monitoring tool generates an alert (e.g., low disk space, service down).
2. Incident Trigger: The alert triggers an incident in Callgoose SQIBS.
3. Auto-Remediation: Callgoose SQIBS triggers a pre-configured PowerShell script that connects to the Windows Server via SSH and resolves the issue (e.g., restarting a service or clearing logs).

For more details on setting up monitoring tools and incident auto-remediation with Callgoose SQIBS, refer to Prometheus - Incident Auto Remediation Documentation.

Final Notes

• Ensure PowerShell 7.x is the default shell for SSH to ensure smooth automation.
• Enable and verify SSH logging if troubleshooting is required.
• Use passwordless SSH connections to avoid manual intervention during incident remediation.