ghaymah-exam-mahmoud-secops/q5-ransomware-response/backup-recovery-strategy.md

Ghaymah Backup & Recovery Strategy

RPO and RTO Targets

Metric	Definition	Target for Ghaymah
RPO (Recovery Point Objective)	Maximum acceptable data loss (how old can a backup be?)	1 hour — automatic snapshots every hour
RTO (Recovery Time Objective)	Maximum acceptable downtime (how fast must we recover?)	4 hours — from incident declaration to restored service

---

The 3-2-1 Backup Rule

The 3-2-1 rule is a commonly used backup best practice:

3 — Keep 3 copies of your data
2 — Store them on 2 different types of media/storage
1 — Keep 1 copy offsite (geographically separate location)

How Ghaymah Implements 3-2-1

graph TD
    subgraph Copy 1 - Primary
        LIVE["💻 Production Instance"]
        BS["🗄️ SSD Block Storage\n(Attached Volume)"]
        LIVE --> BS
    end

    subgraph Copy 2 - Cloud Backup
        OS["☁️ Ghaymah Object Storage\n(Separate Storage Tier)"]
        BS -->|"Automated Hourly Backup\nRetention: 7 days hourly, 30 days daily"| OS
    end

    subgraph Copy 3 - Offsite Remote
        OFF["🌍 Cold/Archive Storage\n(e.g., Secondary Regional Datacenter)"]
        OS -->|"Daily Export\nRetention: 90 days"| OFF
    end

---

Backup Schedule

Backup Type	Frequency	Retention	Storage Location
Ghaymah Object Storage backup	Every 1 hour	7 days	Ghaymah Object Storage (separate from Block Storage)
Daily backup	Every 24 hours at 02:00	30 days	Ghaymah Object Storage
Weekly backup	Every Sunday at 03:00	90 days	Offsite / remote region
Pre-deployment backup	Before every deployment	30 days	Ghaymah Object Storage

---

Automated Snapshot Script

#!/bin/bash
# backup-snapshot.sh — Run via cron every hour

VOLUME_ID="vol-xxxx-replace-with-actual"
SNAPSHOT_PREFIX="auto-backup"
RETENTION_DAYS=7
DATE=$(date +%Y%m%d-%H%M)

echo "[$(date)] Creating snapshot for volume $VOLUME_ID"

# Create snapshot (replace with actual Ghaymah CLI command)
ghaymah-cli storage snapshot create \
  --volume-id "$VOLUME_ID" \
  --name "${SNAPSHOT_PREFIX}-${DATE}"

# Clean up snapshots older than retention window
ghaymah-cli storage snapshot list --volume-id "$VOLUME_ID" \
  | awk -v cutoff="$(date -d "$RETENTION_DAYS days ago" +%s)" \
    '$3 < cutoff {print $1}' \
  | xargs -I{} ghaymah-cli storage snapshot delete --snapshot-id {}

echo "[$(date)] Snapshot created: ${SNAPSHOT_PREFIX}-${DATE}"

Add to crontab:

# Run backup every hour at minute 0
0 * * * * /opt/scripts/backup-snapshot.sh >> /var/log/backup.log 2>&1

---

Recovery Procedure

Standard Recovery (planned or scheduled)

# 1. List available snapshots
ghaymah-cli storage snapshot list --volume-id vol-xxxx

# 2. Create new volume from snapshot
ghaymah-cli storage volume create-from-snapshot \
  --snapshot-id snap-xxxx \
  --name recovery-$(date +%Y%m%d)

# 3. Detach old volume (if still attached)
ghaymah-cli storage volume detach --instance-id inst-xxxx --volume-id vol-old

# 4. Attach new volume
ghaymah-cli storage volume attach --instance-id inst-xxxx --volume-id vol-new

# 5. Mount and verify
sudo mount /dev/sdb /mnt/data
ls -la /mnt/data

Emergency Recovery (ransomware or corruption)

1. Use emergency-plan.md for first 60 minutes.
2. Identify last clean snapshot from before the incident timestamp.
3. Never mount suspicious volumes to production instances.
4. Always verify restored data integrity before going live.

---

Backup Integrity Verification

Monthly verification test:

#!/bin/bash
# verify-backup.sh — Monthly DR drill

echo "=== Backup Verification Test ==="
echo "Date: $(date)"

# 1. Identify latest snapshot
LATEST_SNAP=$(ghaymah-cli storage snapshot list --volume-id vol-xxxx \
  | sort -k3 -r | head -1 | awk '{print $1}')

# 2. Create test restore volume
ghaymah-cli storage volume create-from-snapshot \
  --snapshot-id "$LATEST_SNAP" \
  --name test-restore-$(date +%Y%m%d)

# 3. Attach to test instance (never production)
ghaymah-cli storage volume attach --instance-id inst-test --volume-id vol-test

# 4. Verify expected file structure
EXPECTED_COUNT=$(cat /opt/backup-manifest/expected-file-count.txt)
ACTUAL_COUNT=$(find /mnt/test-restore -type f | wc -l)

echo "Expected files: $EXPECTED_COUNT"
echo "Actual files:   $ACTUAL_COUNT"

if [ "$ACTUAL_COUNT" -ge "$EXPECTED_COUNT" ]; then
  echo "[PASS] Backup integrity verified"
else
  echo "[FAIL] File count mismatch — alert secops team"
fi

# 5. Cleanup test volume
ghaymah-cli storage volume detach --instance-id inst-test --volume-id vol-test
ghaymah-cli storage volume delete --volume-id vol-test