ghaymah-exam-mahmoud-secops/q4-siem/deploy-guide.md

Lightweight SIEM Deployment Guide on Ghaymah Cloud Infrastructure

Overview

This guide explains how to deploy the siem.py log analyzer and dashboard.html on Ghaymah cloud infrastructure, using Block Storage to persist logs and alert data.

---

Architecture

graph TD
    BS["🗄️ Block Storage Volume\n/mnt/logs"]
    BS --> AL["/mnt/logs/auth.log"]
    BS --> NL["/mnt/logs/nginx/access.log"]
    BS --> AP["/mnt/logs/app/application.log"]

    AL & NL & AP --> SIEM["⚙️ siem.py\nsystemd service"]

    SIEM -->|"writes every 30s"| AJ["📄 alerts.json\n/var/www/siem/"]

    AJ -->|"served by Nginx"| DASH["🖥️ dashboard.html\nNear Real-Time UI"]

---

Step 1: Create and Attach Block Storage

In the Ghaymah Cloud Console:

1. Go to Storage → Block Storage → Create Volume
2. Set name: siem-logs-volume
3. Set size: 50 GB (adjust based on log retention needs)
4. Attach to your cloud instance

Then format and mount the volume:

# Find the new volume (usually /dev/sdb or /dev/vdb)
lsblk

# Format the volume
sudo mkfs.ext4 /dev/sdb

# Create mount point
sudo mkdir -p /mnt/logs

# Mount the volume
sudo mount /dev/sdb /mnt/logs

# Persist mount across reboots — use UUID to avoid /dev/sdX changes after reboot
# First find the UUID:
sudo blkid /dev/sdb
# Then replace UUID=xxxx with the actual value shown:
echo 'UUID=xxxx-xxxx /mnt/logs ext4 defaults,nofail 0 2' | sudo tee -a /etc/fstab

# Create log directories
sudo mkdir -p /mnt/logs/nginx /mnt/logs/app
sudo chown -R $USER:$USER /mnt/logs

Encryption: Enable volume encryption during Block Storage creation in the Ghaymah console before writing any data to the volume.

---

Step 2: Configure Services to Write Logs to Block Storage

Nginx

# Edit nginx.conf to write access log to block storage
sudo nano /etc/nginx/nginx.conf
# Change:
# access_log /var/log/nginx/access.log;
# To:
# access_log /mnt/logs/nginx/access.log;

sudo systemctl reload nginx

Application (Docker)

# docker-compose.yml
services:
  api:
    volumes:
      - /mnt/logs/app:/var/log/app
    environment:
      LOG_PATH: /var/log/app/application.log

Secrets management note: For production deployments, avoid plain .env files on disk. Prefer Kubernetes Secrets (mounted as env vars or files) or a cloud secret manager (e.g., HashiCorp Vault, AWS Secrets Manager, or your cloud provider's equivalent). .env files are acceptable for local development but should never be committed to Git.

SSH Auth Log Forwarding

# Create the auth log file with correct permissions for rsyslog
sudo touch /mnt/logs/auth.log
sudo chown syslog:adm /mnt/logs/auth.log
sudo chmod 640 /mnt/logs/auth.log

# Direct rsyslog to write auth logs to block storage
echo 'auth,authpriv.* /mnt/logs/auth.log' | sudo tee /etc/rsyslog.d/99-siem.conf
sudo systemctl restart rsyslog

---

Step 3: Deploy siem.py as a Systemd Service

# Install Python 3
sudo apt-get update && sudo apt-get install -y python3

# Create a shared group so both siem.py (ubuntu) and nginx (www-data) can access alerts.json
sudo groupadd siem
sudo usermod -aG siem ubuntu
sudo usermod -aG siem www-data

# Copy SIEM files
sudo mkdir -p /opt/siem /var/www/siem
sudo cp q4-siem/siem.py /opt/siem/siem.py
sudo cp q4-siem/dashboard.html /var/www/siem/dashboard.html

# Set permissions: ubuntu writes, www-data reads via shared siem group
sudo touch /var/www/siem/alerts.json
sudo chown ubuntu:siem /var/www/siem/alerts.json
sudo chmod 640 /var/www/siem/alerts.json

# Edit log paths in siem.py to match block storage
sudo nano /opt/siem/siem.py
# Set:
# LOG_SOURCES = {
#     "auth":  "/mnt/logs/auth.log",
#     "nginx": "/mnt/logs/nginx/access.log",
#     "app":   "/mnt/logs/app/application.log",
# }
# OUTPUT_FILE = "/var/www/siem/alerts.json"

Create the systemd service:

# /etc/systemd/system/siem.service
[Unit]
Description=Ghaymah SIEM Log Analyzer
After=network.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/opt/siem
ExecStart=/usr/bin/python3 /opt/siem/siem.py
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable siem
sudo systemctl start siem
sudo systemctl status siem

---

Step 4: Serve the Dashboard via Nginx

# /etc/nginx/sites-available/siem
server {
    listen 8443 ssl;
    server_name siem.ghaymah.systems;

    ssl_certificate     /etc/letsencrypt/live/siem.ghaymah.systems/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/siem.ghaymah.systems/privkey.pem;

    root /var/www/siem;
    index dashboard.html;

    # Restrict access to internal network only
    allow 10.0.0.0/8;
    allow 192.168.0.0/16;
    deny all;

    # alerts.json is served to the dashboard browser client.
    # The parent allow/deny block already restricts access to internal IPs only.
    location = /alerts.json {
        allow 10.0.0.0/8;
        allow 192.168.0.0/16;
        deny all;
        add_header Content-Type application/json;
    }

    location / {
        try_files $uri $uri/ =404;
    }
}

sudo ln -s /etc/nginx/sites-available/siem /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

---

Step 5: Log Rotation

# /etc/logrotate.d/siem-logs
/mnt/logs/*.log
/mnt/logs/app/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 640 syslog adm
    postrotate
        systemctl restart rsyslog
    endscript
}

/mnt/logs/nginx/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 640 www-data adm
    postrotate
        systemctl reload nginx
    endscript
}

---

Verification

# Check SIEM is running
sudo systemctl status siem

# Check alerts output
cat /var/www/siem/alerts.json | python3 -m json.tool | head -30

# View SIEM logs
sudo journalctl -u siem -f

# Test dashboard
curl -k https://siem.ghaymah.systems:8443/dashboard.html

---

Security Notes

Control	Implementation
Dashboard access	Restricted to internal IP range (10.0.0.0/8)
HTTPS	TLS required, no HTTP access
alerts.json access	Internal IPs only via allow 10.0.0.0/8; deny all; inside location = /alerts.json
File permissions	alerts.json owned ubuntu:siem, mode 640 — owner writes, group (nginx) reads, others none
Block storage	Encrypted at rest (enable during volume creation in Ghaymah console)
Audit logs	All SIEM activity written to systemd journal

---

Future Integrations

The SIEM is designed to grow. The following integrations can be added as next steps:

Integration	How
AlertManager webhook	Extend siem.py to POST critical alerts to a webhook endpoint. Pairs directly with the Prometheus alert rules in q2-attack-simulation/alert-rule.yml.
Slack notifications	Send alerts to a #security-alerts channel via Slack Incoming Webhooks when severity is CRITICAL or HIGH.
PagerDuty / on-call	Route CRITICAL alerts to PagerDuty for 24/7 on-call escalation using the Events API.
Email alerts	Use smtplib in siem.py to send summary emails to the security team at the end of each scan cycle.
Cloud monitoring export	Ship alerts.json to a cloud-native monitoring service (e.g., Ghaymah monitoring, Datadog, or CloudWatch) for long-term trend analysis and dashboards.
Log shipping to SIEM platform	Forward structured logs to a dedicated SIEM platform (e.g., Elastic/OpenSearch) for retention and correlation across multiple services.

---

Backup Strategy

Block Storage volumes are the persistence layer for all logs and alert data. Schedule regular snapshots to protect against data loss.

Recommended snapshot schedule:

Frequency	Retention	Purpose
Daily	7 days	Short-term recovery from accidental deletion or corruption
Weekly	4 weeks	Recovery from persistent misconfiguration issues
Monthly	3 months	Compliance and long-term trend analysis

How to enable in Ghaymah console:

1. Go to Storage → Block Storage → siem-logs-volume
2. Navigate to Snapshots → Create Snapshot Policy
3. Set a daily schedule with 7-day retention
4. Enable email notification on snapshot failure

Restore procedure (monthly test recommended):

# Restore a snapshot to a new volume in the Ghaymah console,
# then mount it temporarily to verify data integrity:
sudo mkdir -p /mnt/logs-restore
sudo mount /dev/sdc /mnt/logs-restore
ls -lh /mnt/logs-restore/
sudo umount /mnt/logs-restore

Restore tests should be performed monthly to confirm backups are valid and recovery time is within acceptable limits.