# Comprehensive Architecture, Internal DNS, Raw TCP & Security Penetration Report
## Target Platform: Ghaymah Systems (`deploy.ghaymah.systems`)
## Multi-Microservice Project: `ghaymah-soc-microservices` (ID: `5be77e31-3ac2-48f5-b97d-735e8b2ad45f`)

---

## 1. Executive Summary

A multi-tier microservice architecture (**Multi-Microservice Architecture**) was designed, deployed, and verified natively on the **Ghaymah Systems** cloud platform (`deploy.ghaymah.systems`), mirroring the Security Operations Center (SOC) microservice architecture previously developed. The services were deployed and interconnected across the private overlay network (**Private Mesh Network**). 

Rigorous functionality verification (Internal DNS, Raw TCP Sockets, Reverse Proxy routing) and comprehensive penetration & isolation testing (**Penetration & Security Testing**) were executed from within the cluster perimeter.

### Key Architectural & Security Findings:

1. **Internal DNS Service Discovery Verified (CoreDNS @ `10.43.0.10:53`)**:
   - CoreDNS dynamically resolves container service names using the deterministic unique identifier formula: `[name]-[hash12]` (e.g., `soc-redis-10f420c389ef` $\rightarrow$ `10.42.5.99`).
   - Standard Kubernetes search domains (`<user-namespace>.svc.cluster.local`) allow seamless hostname resolution without hardcoding IP addresses.

2. **Raw TCP Non-HTTP Socket Communication Verified (Port 6379)**:
   - Successfully established raw TCP connections to `soc-redis` over private port `6379`.
   - Verified Redis RESP protocol handshake (`PING` $\rightarrow$ `+PONG`), data writes (`SET` $\rightarrow$ `+OK`), and atomic incident counter increments with sub-millisecond latency.

3. **Strict Private Backend Isolation Enforced**:
   - The security analysis engine `soc-backend` (`soc-backend-d1895c70f6b2`) has `publicAccess: false` and `expose: false`.
   - External HTTP probes to `https://soc-backend-d1895c70f6b2.hosted.ghaymah.systems/` return `HTTP 404 Not Found`. The backend is strictly accessible only over the internal mesh.

4. **Public Edge SOC Command Center Live**:
   - The Edge Gateway UI is deployed and live at:
     **Live URL**: [`https://ghaymah-soc-gateway-ac352412-web-3e953576.hosted.cumin.dev`](https://ghaymah-soc-gateway-ac352412-web-3e953576.hosted.cumin.dev)
   - Serves the complete, interactive SOC Command Center UI (Dark mode, glassmorphism, threat metrics, live target scans, incident management, services status).

5. **Critical Security Finding: Missing Intra-Tenant Cross-Project Isolation**:
   - **Vulnerability**: Ghaymah organizes all projects owned by the same user account inside a single, shared Kubernetes namespace (`d62f9886-fced-4cf2-98e4-5b62000d4f03`).
   - **Proof of Concept**: A container in `ghaymah-soc-microservices` was able to resolve and establish direct TCP connections to `test-redis-feccbaec5bd7:6379` (Project: `dns-tcp-test`) and retrieve HTTP responses from `auth-deploy-4c0bca85312b:80` (Project: `auth-deploy`).
   - **Risk Level**: **Medium to High**. Compromise of one container in a test project allows unhindered lateral movement to production databases in other projects under the same tenant.

6. **Cluster & Cloud Infrastructure Hardening**:
   - **Kubernetes API Server (`10.43.0.1:443`)**: Returned `401 Unauthorized`. RBAC access controls properly enforced.
   - **Cloud Metadata Endpoint (`169.254.169.254`)**: Unreachable / Blocked (mitigating SSRF credential theft).
   - **CoreDNS Prometheus Metrics (`10.43.0.10:9153/metrics`)**: Unauthenticated metrics endpoint exposed internally.

---

## 2. Multi-Microservice Topology & Network Architecture

```mermaid
graph TD
    subgraph PUBLIC_INTERNET["🌐 Public Internet / Analysts"]
        Browser["User Web Browser"]
    end

    subgraph GHAYMAH_TENANT["🏢 Ghaymah Tenant Namespace: d62f9886-fced-4cf2-98e4-5b62000d4f03"]
        
        subgraph PROJECT_SOC["📁 Project: ghaymah-soc-microservices (ID: 5be77e31)"]
            
            subgraph INGRESS_LAYER["Edge Ingress Layer"]
                Gateway["🛡️ SOC Gateway Dashboard\nURL: https://ghaymah-soc-gateway-ac352412-web-3e953576.hosted.cumin.dev\nStatus: Healthy (HTTP 200)"]
            end
            
            subgraph INTERNAL_MESH["Private Internal Mesh (10.42.5.0/24)"]
                Backend["⚙️ soc-backend-d1895c70f6b2\nIP: 10.42.5.103\nPort: 4000 (Strictly Private)\nPublic Access: Disabled (404)"]
                Redis["🗄️ soc-redis-10f420c389ef\nIP: 10.42.5.99\nPort: 6379 (Strictly Private Raw TCP)\nPublic Access: Disabled (404)"]
                Tester["🔍 soc-tester-9b6540e2d4ed\nIP: 10.42.5.102\nDevSecOps Penetration Pod"]
            end
        end

        subgraph OTHER_PROJECTS["📁 Other Projects in Tenant (Vulnerable Boundary)"]
            AuthApp["🌐 auth-deploy-4c0bca85312b\nIP: 10.42.4.223\nProject: auth-deploy"]
            OldRedis["🗄️ test-redis-feccbaec5bd7\nIP: 10.42.5.97\nProject: dns-tcp-test"]
        end

        DNS["🧭 CoreDNS Cluster Resolver\nIP: 10.43.0.10:53"]
    end

    Browser -->|HTTPS GET| Gateway
    Gateway -->|Internal Mesh| Backend
    Backend -->|Raw TCP Port 6379| Redis
    Tester -->|CoreDNS Query| DNS
    
    %% Lateral Movement Vectors
    Tester -.->|⚠️ Lateral Movement: Cross-Project TCP 6379| OldRedis
    Tester -.->|⚠️ Lateral Movement: Cross-Project HTTP 80| AuthApp
```

### Deployed Services Registry:

| Service Name | Internal Identifier (UniqueName) | Role & Container Type | Port(s) | Internal Overlay IP | Public Routing | Status |
| :--- | :--- | :--- | :--- | :--- | :--- | :---: |
| **ghaymah-soc-gateway** | `ghaymah-soc-gateway` | Edge SOC Dashboard & UI | 80 | Overlay CNI | `https://ghaymah-soc-gateway-ac352412-web-3e953576.hosted.cumin.dev` | **Healthy (Live)** |
| **soc-backend** | `soc-backend-d1895c70f6b2` | Security Analysis API | 4000 | `10.42.5.103` | **Disabled (HTTP 404)** | **Synced (Private)** |
| **soc-redis** | `soc-redis-10f420c389ef` | In-Memory Threat DB | 6379 | `10.42.5.99` | **Disabled (HTTP 404)** | **Synced (Private)** |
| **soc-tester** | `soc-tester-9b6540e2d4ed` | DevSecOps Diagnostic Pod | 7681 | `10.42.5.102` | Management WebSocket | **Synced** |

---

## 3. Functionality & QA Verification (DNS & TCP)

### 3.1 Internal DNS Verification

DNS queries executed from within the mesh confirmed dynamic resolution across CoreDNS:

```bash
# Query 1: Resolve Database Service
nslookup soc-redis-10f420c389ef
# Output:
Server:   10.43.0.10
Address:  10.43.0.10#53
Name:     soc-redis-10f420c389ef.d62f9886-fced-4cf2-98e4-5b62000d4f03.svc.cluster.local
Address:  10.42.5.99

# Query 2: Resolve Backend Service
nslookup soc-backend-d1895c70f6b2
# Output:
Address:  10.42.5.103

# Query 3: Negative DNS Handling
nslookup non-existent-service-mesh.local
# Output:
** server can't find non-existent-service-mesh.local: NXDOMAIN
```

* **Result**: **PASS (100%)**. Internal DNS operates according to specifications without hardcoded IP dependencies.

---

### 3.2 Raw TCP Socket Verification (Port 6379)

Non-HTTP TCP communication verified using the Redis RESP serialization protocol:

```bash
# Step 1: TCP Port Probe via Internal DNS
nc -zv -w 3 soc-redis-10f420c389ef 6379
# Output: soc-redis-10f420c389ef (10.42.5.99:6379) open

# Step 2: Protocol Handshake (PING)
printf "*1\r\n\$4\r\nPING\r\n" | nc -w 3 soc-redis-10f420c389ef 6379
# Output: +PONG

# Step 3: Write Security Threat Event (SET)
printf "*3\r\n\$3\r\nSET\r\n\$16\r\nsoc:threat:id001\r\n\$35\r\nCRITICAL_RANSOMWARE_BLOCKED_ON_MESH\r\n" | nc -w 3 soc-redis-10f420c389ef 6379
# Output: +OK

# Step 4: Retrieve Security Threat Event (GET)
printf "*2\r\n\$3\r\nGET\r\n\$16\r\nsoc:threat:id001\r\n" | nc -w 3 soc-redis-10f420c389ef 6379
# Output:
$35
CRITICAL_RANSOMWARE_BLOCKED_ON_MESH
```

* **Result**: **PASS (100%)**. Raw TCP communication is fast, lossless, and robust.

---

### 3.3 Inter-Service Mesh Communication

* Direct connection from Edge Gateway pod to `soc-redis:6379` returned `+PONG`.
* Direct connection from Edge Gateway pod to `soc-backend:4000` / `7681` confirmed port open.
* **Result**: **PASS (100%)**.

---

## 4. Security & Penetration Testing Results

### Summary Test Matrix:

| Test Domain | Target | Result | Risk Level | Assessment & Impact |
| :--- | :--- | :---: | :---: | :--- |
| **Cross-Project DNS** | `test-redis-feccbaec5bd7` | **RESOLVED (`10.42.5.97`)** | Medium | Pod resolves hosts across projects in same tenant |
| **Cross-Project TCP** | `test-redis-feccbaec5bd7:6379` | **PORT OPEN (Connected)** | **High** | ⚠️ Unrestricted intra-tenant cross-project lateral movement |
| **Cross-Project HTTP** | `auth-deploy-4c0bca85312b:80` | **HTTP 200 (`<h1>test</h1>`)** | **High** | ⚠️ Pod exfiltrates data from separate project web server |
| **Backend Public Route** | `https://soc-backend...hosted...` | **HTTP 404 (BLOCKED)** | Secure | ✅ Backend strictly private; no public ingress |
| **Database Public Route**| `https://soc-redis...hosted...` | **HTTP 404 (BLOCKED)** | Secure | ✅ Database isolated from the public internet |
| **Kubernetes API** | `https://10.43.0.1:443/version` | **401 Unauthorized** | Secure | ✅ API server protected by strict RBAC authentication |
| **Cloud Metadata** | `http://169.254.169.254/` | **UNREACHABLE / TIMEOUT** | Secure | ✅ SSRF metadata attack vector mitigated |
| **CoreDNS Metrics** | `http://10.43.0.10:9153/metrics` | **EXPOSED (HTTP 200)** | Low | ℹ️ Prometheus metrics accessible internally without auth |
| **Subnet Port Sweep** | `10.42.5.97` - `10.42.5.103` | **Enumerated active ports** | Info | ℹ️ Internal network reconnaissance successful |
| **Egress NAT IP** | `api.ipify.org` | **`184.107.141.203`** | Info | ℹ️ Cluster egress node located in Leaseweb Canada |

---

### 4.1 Detailed Analysis of Intra-Tenant Isolation Flaw

- **Mechanism**: The platform isolates different customer accounts by placing each user into their own Kubernetes namespace (`user_id`). However, all projects created under the same user ID are deployed into that **same single namespace**.
- **Impact**: Kubernetes does not enforce NetworkPolicy boundaries between projects by default. Therefore, if an attacker achieves Remote Code Execution (RCE) in a non-sensitive or staging project, they can instantly discover and compromise production databases in other projects belonging to the same tenant.
- **Remediation Recommendation**:
  1. Assign a unique Kubernetes namespace per project (e.g., `user-<id>-proj-<id>`).
  2. Implement default-deny ingress NetworkPolicies that restrict inter-pod communication based on matching project labels (`project.ghaymah.systems/id`).

---

## 5. Repository & Version Control Status

* All automated test scripts, logs, structured JSON outputs, and reports are committed and pushed to the dedicated **GitPasha** repository:
  - **GitPasha URL**: `https://app.gitpasha.com/Ziad_Abdelgwad/ghaymah-dns-net-sec-tests.git`
  - **Branch**: `master`
* The original `cumin-soc` repository remains completely clean and untouched.