164 أسطر
4.6 KiB
Markdown
164 أسطر
4.6 KiB
Markdown
# Ghaymah Backup & Recovery Strategy
|
|
|
|
## RPO and RTO Targets
|
|
|
|
| Metric | Definition | Target for Ghaymah |
|
|
|---|---|---|
|
|
| **RPO** (Recovery Point Objective) | Maximum acceptable data loss (how old can a backup be?) | **1 hour** — automatic snapshots every hour |
|
|
| **RTO** (Recovery Time Objective) | Maximum acceptable downtime (how fast must we recover?) | **4 hours** — from incident declaration to restored service |
|
|
|
|
---
|
|
|
|
## The 3-2-1 Backup Rule
|
|
|
|
The **3-2-1 rule** is a commonly used backup best practice:
|
|
|
|
```
|
|
3 — Keep 3 copies of your data
|
|
2 — Store them on 2 different types of media/storage
|
|
1 — Keep 1 copy offsite (geographically separate location)
|
|
```
|
|
|
|
### How Ghaymah Implements 3-2-1
|
|
|
|
```mermaid
|
|
graph TD
|
|
subgraph Copy 1 - Primary
|
|
LIVE["💻 Production Instance"]
|
|
BS["🗄️ SSD Block Storage\n(Attached Volume)"]
|
|
LIVE --> BS
|
|
end
|
|
|
|
subgraph Copy 2 - Cloud Backup
|
|
OS["☁️ Ghaymah Object Storage\n(Separate Storage Tier)"]
|
|
BS -->|"Automated Hourly Backup\nRetention: 7 days hourly, 30 days daily"| OS
|
|
end
|
|
|
|
subgraph Copy 3 - Offsite Remote
|
|
OFF["🌍 Cold/Archive Storage\n(e.g., Secondary Regional Datacenter)"]
|
|
OS -->|"Daily Export\nRetention: 90 days"| OFF
|
|
end
|
|
```
|
|
|
|
---
|
|
|
|
## Backup Schedule
|
|
|
|
| Backup Type | Frequency | Retention | Storage Location |
|
|
|---|---|---|---|
|
|
| Ghaymah Object Storage backup | Every 1 hour | 7 days | Ghaymah Object Storage (separate from Block Storage) |
|
|
| Daily backup | Every 24 hours at 02:00 | 30 days | Ghaymah Object Storage |
|
|
| Weekly backup | Every Sunday at 03:00 | 90 days | Offsite / remote region |
|
|
| Pre-deployment backup | Before every deployment | 30 days | Ghaymah Object Storage |
|
|
|
|
---
|
|
|
|
## Automated Snapshot Script
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# backup-snapshot.sh — Run via cron every hour
|
|
|
|
VOLUME_ID="vol-xxxx-replace-with-actual"
|
|
SNAPSHOT_PREFIX="auto-backup"
|
|
RETENTION_DAYS=7
|
|
DATE=$(date +%Y%m%d-%H%M)
|
|
|
|
echo "[$(date)] Creating snapshot for volume $VOLUME_ID"
|
|
|
|
# Create snapshot (replace with actual Ghaymah CLI command)
|
|
ghaymah-cli storage snapshot create \
|
|
--volume-id "$VOLUME_ID" \
|
|
--name "${SNAPSHOT_PREFIX}-${DATE}"
|
|
|
|
# Clean up snapshots older than retention window
|
|
ghaymah-cli storage snapshot list --volume-id "$VOLUME_ID" \
|
|
| awk -v cutoff="$(date -d "$RETENTION_DAYS days ago" +%s)" \
|
|
'$3 < cutoff {print $1}' \
|
|
| xargs -I{} ghaymah-cli storage snapshot delete --snapshot-id {}
|
|
|
|
echo "[$(date)] Snapshot created: ${SNAPSHOT_PREFIX}-${DATE}"
|
|
```
|
|
|
|
Add to crontab:
|
|
```bash
|
|
# Run backup every hour at minute 0
|
|
0 * * * * /opt/scripts/backup-snapshot.sh >> /var/log/backup.log 2>&1
|
|
```
|
|
|
|
---
|
|
|
|
## Recovery Procedure
|
|
|
|
### Standard Recovery (planned or scheduled)
|
|
|
|
```bash
|
|
# 1. List available snapshots
|
|
ghaymah-cli storage snapshot list --volume-id vol-xxxx
|
|
|
|
# 2. Create new volume from snapshot
|
|
ghaymah-cli storage volume create-from-snapshot \
|
|
--snapshot-id snap-xxxx \
|
|
--name recovery-$(date +%Y%m%d)
|
|
|
|
# 3. Detach old volume (if still attached)
|
|
ghaymah-cli storage volume detach --instance-id inst-xxxx --volume-id vol-old
|
|
|
|
# 4. Attach new volume
|
|
ghaymah-cli storage volume attach --instance-id inst-xxxx --volume-id vol-new
|
|
|
|
# 5. Mount and verify
|
|
sudo mount /dev/sdb /mnt/data
|
|
ls -la /mnt/data
|
|
```
|
|
|
|
### Emergency Recovery (ransomware or corruption)
|
|
|
|
1. Use emergency-plan.md for first 60 minutes.
|
|
2. Identify last clean snapshot from **before** the incident timestamp.
|
|
3. Never mount suspicious volumes to production instances.
|
|
4. Always verify restored data integrity before going live.
|
|
|
|
---
|
|
|
|
## Backup Integrity Verification
|
|
|
|
Monthly verification test:
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# verify-backup.sh — Monthly DR drill
|
|
|
|
echo "=== Backup Verification Test ==="
|
|
echo "Date: $(date)"
|
|
|
|
# 1. Identify latest snapshot
|
|
LATEST_SNAP=$(ghaymah-cli storage snapshot list --volume-id vol-xxxx \
|
|
| sort -k3 -r | head -1 | awk '{print $1}')
|
|
|
|
# 2. Create test restore volume
|
|
ghaymah-cli storage volume create-from-snapshot \
|
|
--snapshot-id "$LATEST_SNAP" \
|
|
--name test-restore-$(date +%Y%m%d)
|
|
|
|
# 3. Attach to test instance (never production)
|
|
ghaymah-cli storage volume attach --instance-id inst-test --volume-id vol-test
|
|
|
|
# 4. Verify expected file structure
|
|
EXPECTED_COUNT=$(cat /opt/backup-manifest/expected-file-count.txt)
|
|
ACTUAL_COUNT=$(find /mnt/test-restore -type f | wc -l)
|
|
|
|
echo "Expected files: $EXPECTED_COUNT"
|
|
echo "Actual files: $ACTUAL_COUNT"
|
|
|
|
if [ "$ACTUAL_COUNT" -ge "$EXPECTED_COUNT" ]; then
|
|
echo "[PASS] Backup integrity verified"
|
|
else
|
|
echo "[FAIL] File count mismatch — alert secops team"
|
|
fi
|
|
|
|
# 5. Cleanup test volume
|
|
ghaymah-cli storage volume detach --instance-id inst-test --volume-id vol-test
|
|
ghaymah-cli storage volume delete --volume-id vol-test
|
|
```
|