# Ghaymah Backup & Recovery Strategy

## RPO and RTO Targets

| Metric | Definition | Target for Ghaymah |
|---|---|---|
| **RPO** (Recovery Point Objective) | Maximum acceptable data loss (how old can a backup be?) | **1 hour** — automatic snapshots every hour |
| **RTO** (Recovery Time Objective) | Maximum acceptable downtime (how fast must we recover?) | **4 hours** — from incident declaration to restored service |

---

## The 3-2-1 Backup Rule

The **3-2-1 rule** is a commonly used backup best practice:

```
3 — Keep 3 copies of your data
2 — Store them on 2 different types of media/storage
1 — Keep 1 copy offsite (geographically separate location)
```

### How Ghaymah Implements 3-2-1

```mermaid
graph TD
    subgraph Copy 1 - Primary
        LIVE["💻 Production Instance"]
        BS["🗄️ SSD Block Storage\n(Attached Volume)"]
        LIVE --> BS
    end

    subgraph Copy 2 - Cloud Backup
        OS["☁️ Ghaymah Object Storage\n(Separate Storage Tier)"]
        BS -->|"Automated Hourly Backup\nRetention: 7 days hourly, 30 days daily"| OS
    end

    subgraph Copy 3 - Offsite Remote
        OFF["🌍 Cold/Archive Storage\n(e.g., Secondary Regional Datacenter)"]
        OS -->|"Daily Export\nRetention: 90 days"| OFF
    end
```

---

## Backup Schedule

| Backup Type | Frequency | Retention | Storage Location |
|---|---|---|---|
| Ghaymah Object Storage backup | Every 1 hour | 7 days | Ghaymah Object Storage (separate from Block Storage) |
| Daily backup | Every 24 hours at 02:00 | 30 days | Ghaymah Object Storage |
| Weekly backup | Every Sunday at 03:00 | 90 days | Offsite / remote region |
| Pre-deployment backup | Before every deployment | 30 days | Ghaymah Object Storage |

---

## Automated Snapshot Script

```bash
#!/bin/bash
# backup-snapshot.sh — Run via cron every hour

VOLUME_ID="vol-xxxx-replace-with-actual"
SNAPSHOT_PREFIX="auto-backup"
RETENTION_DAYS=7
DATE=$(date +%Y%m%d-%H%M)

echo "[$(date)] Creating snapshot for volume $VOLUME_ID"

# Create snapshot (replace with actual Ghaymah CLI command)
ghaymah-cli storage snapshot create \
  --volume-id "$VOLUME_ID" \
  --name "${SNAPSHOT_PREFIX}-${DATE}"

# Clean up snapshots older than retention window
ghaymah-cli storage snapshot list --volume-id "$VOLUME_ID" \
  | awk -v cutoff="$(date -d "$RETENTION_DAYS days ago" +%s)" \
    '$3 < cutoff {print $1}' \
  | xargs -I{} ghaymah-cli storage snapshot delete --snapshot-id {}

echo "[$(date)] Snapshot created: ${SNAPSHOT_PREFIX}-${DATE}"
```

Add to crontab:
```bash
# Run backup every hour at minute 0
0 * * * * /opt/scripts/backup-snapshot.sh >> /var/log/backup.log 2>&1
```

---

## Recovery Procedure

### Standard Recovery (planned or scheduled)

```bash
# 1. List available snapshots
ghaymah-cli storage snapshot list --volume-id vol-xxxx

# 2. Create new volume from snapshot
ghaymah-cli storage volume create-from-snapshot \
  --snapshot-id snap-xxxx \
  --name recovery-$(date +%Y%m%d)

# 3. Detach old volume (if still attached)
ghaymah-cli storage volume detach --instance-id inst-xxxx --volume-id vol-old

# 4. Attach new volume
ghaymah-cli storage volume attach --instance-id inst-xxxx --volume-id vol-new

# 5. Mount and verify
sudo mount /dev/sdb /mnt/data
ls -la /mnt/data
```

### Emergency Recovery (ransomware or corruption)

1. Use emergency-plan.md for first 60 minutes.
2. Identify last clean snapshot from **before** the incident timestamp.
3. Never mount suspicious volumes to production instances.
4. Always verify restored data integrity before going live.

---

## Backup Integrity Verification

Monthly verification test:

```bash
#!/bin/bash
# verify-backup.sh — Monthly DR drill

echo "=== Backup Verification Test ==="
echo "Date: $(date)"

# 1. Identify latest snapshot
LATEST_SNAP=$(ghaymah-cli storage snapshot list --volume-id vol-xxxx \
  | sort -k3 -r | head -1 | awk '{print $1}')

# 2. Create test restore volume
ghaymah-cli storage volume create-from-snapshot \
  --snapshot-id "$LATEST_SNAP" \
  --name test-restore-$(date +%Y%m%d)

# 3. Attach to test instance (never production)
ghaymah-cli storage volume attach --instance-id inst-test --volume-id vol-test

# 4. Verify expected file structure
EXPECTED_COUNT=$(cat /opt/backup-manifest/expected-file-count.txt)
ACTUAL_COUNT=$(find /mnt/test-restore -type f | wc -l)

echo "Expected files: $EXPECTED_COUNT"
echo "Actual files:   $ACTUAL_COUNT"

if [ "$ACTUAL_COUNT" -ge "$EXPECTED_COUNT" ]; then
  echo "[PASS] Backup integrity verified"
else
  echo "[FAIL] File count mismatch — alert secops team"
fi

# 5. Cleanup test volume
ghaymah-cli storage volume detach --instance-id inst-test --volume-id vol-test
ghaymah-cli storage volume delete --volume-id vol-test
```