# Ghaymah Systems — Private Networking & Internal DNS
## Comprehensive QA Functionality & Security Assessment Report

**Execution Date:** 2026-10-12  
**Target Platform:** [Ghaymah Systems](https://deploy.ghaymah.systems/)  
**Environment:** Production Kubernetes Cluster (Leaseweb MTL2 / Canada)  
**Tenant ID:** `d62f9886-fced-4cf2-98e4-5b62000d4f03`  
**Test Project:** `dns-tcp-test` (ID: `3f55ceef-78ae-40c8-a41b-0505db8c59ee`)  
**Assessed By:** DevSecOps & QA Automation Agent (Antigravity IDE)  

---

## 1. Executive Summary

Ghaymah Systems recently released a new update featuring **Private Networking & Internal DNS** within containerized deployments. This audit conducted automated end-to-end functionality verification, QA validation, and defensive security/penetration testing natively on the live Ghaymah platform (`deploy.ghaymah.systems`).

### Key Assessment Results

| Category | Objective | Status | Severity / Result |
| :--- | :--- | :---: | :--- |
| **Internal DNS** | Verify discovery & resolution of containers within same project | **PASS** | CoreDNS (`10.43.0.10`) resolves services via `[app-name]-[hash]` identifiers |
| **Raw TCP** | Verify non-HTTP raw TCP socket communication | **PASS** | Redis protocol (`PING`/`PONG`, `SET`/`GET`) succeeded over port 6379 |
| **Tenant Isolation** | Prevent access to other tenant/cluster infrastructure | **PASS** | Cluster API, host gateways, and cross-tenant domains are strictly blocked |
| **Project Scoping** | Verify network isolation between projects owned by same user | **FINDING** | **MEDIUM/HIGH**: No intra-tenant NetworkPolicies. Cross-project TCP & DNS allowed |
| **Egress & DNS Leaks** | Dual routing verification & external DNS leak testing | **PASS** | Outbound NAT active (`184.107.141.203`), no DNS query leakage |

---

## 2. Architecture & Topology

The Ghaymah infrastructure orchestrates user applications within dedicated Kubernetes namespaces mapped directly to the user's account ID (`d62f9886-fced-4cf2-98e4-5b62000d4f03`). Applications receive internal IP addresses on an overlay network (`10.42.0.0/16`) managed by the CNI.

```mermaid
flowchart TD
    subgraph EXT["🌐 External Network & Internet"]
        User["👨‍💻 Client / Tester"]
        Internet["🌍 Public Internet\nEgress IP: 184.107.141.203"]
    end

    subgraph GHAYMAH_GW["🛡️ Ingress & Edge Proxy"]
        Ingress["Traefik / Ingress Controller\n*.hosted.ghaymah.systems"]
    end

    subgraph TENANT_NS["📦 User Tenant Namespace: d62f9886-fced-4cf2-98e4-5b62000d4f03"]
        direction TB

        subgraph PROJ_A["📁 Project: dns-tcp-test"]
            Term["💻 client-terminal-0f1fe89209a8\nIP: 10.42.5.98\nExposed: Port 7681 (ttyd)"]
            Redis["🗄️ test-redis-feccbaec5bd7\nIP: 10.42.5.97\nPrivate: Port 6379"]
        end

        subgraph PROJ_B["📁 Project: auth-deploy (Separate Project)"]
            AuthApp["🌐 auth-deploy-4c0bca85312b\nIP: 10.42.4.223\nPort 80 (nginx)"]
        end

        DNS["🔍 CoreDNS Service\n10.43.0.10:53"]
    end

    User -->|HTTPS| Ingress
    Ingress -->|Port 7681| Term
    Term -->|1. DNS Query: test-redis-feccbaec5bd7| DNS
    DNS -->|2. Returns 10.42.5.97| Term
    Term -->|3. Raw TCP Socket: Port 6379| Redis
    Term -.->|⚠️ Unrestricted Cross-Project Access| AuthApp
    Term -->|Outbound Egress NAT| Internet
```

### Deployed Workloads on Ghaymah

| Application Name | Unique Identifier | Project | Container Image | Internal IP | Ports | Public Domain | Status |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :---: |
| `client-terminal` | `client-terminal-0f1fe89209a8` | `dns-tcp-test` | `tsl0922/ttyd:alpine` | `10.42.5.98` | 7681 | `client-terminal-0f1fe89209a8.hosted.ghaymah.systems` | **Synced** |
| `test-redis` | `test-redis-feccbaec5bd7` | `dns-tcp-test` | `redis:alpine` | `10.42.5.97` | 6379 | *None (Private Only)* | **Synced** |
| `auth-deploy` | `auth-deploy-4c0bca85312b` | `auth-deploy` | `registry.ghaymah.systems/...` | `10.42.4.223` | 80 | `auth-deploy-4c0bca85312b.hosted.ghaymah.systems` | **Synced** |

---

## 3. Phase 3: Functionality & QA Verification

### 3.1 Container Network Discovery

Inspecting network interfaces and resolver configuration from within `client-terminal`:

- **Network Interfaces (`ip addr`):**
  - `lo`: `127.0.0.1/8`
  - `eth0`: `10.42.5.98/24` (MTU 1250, CNI overlay interface)
- **Routing Table (`ip route`):**
  - `default via 10.42.5.1 dev eth0`
  - `10.42.0.0/16 via 10.42.5.1 dev eth0`
  - `10.42.5.0/24 dev eth0 scope link src 10.42.5.98`
- **DNS Resolver (`/etc/resolv.conf`):**
  ```text
  search d62f9886-fced-4cf2-98e4-5b62000d4f03.svc.cluster.local svc.cluster.local cluster.local cs221.mtl2.ca.leaseweb.net
  nameserver 10.43.0.10
  options ndots:5
  ```

### 3.2 Internal DNS Resolution

Ghaymah assigns unique deterministic names to resources using a SHA-256 hash algorithm:
$$\text{uniqueName} = \text{name} + \text{"-"} + \text{SHA256}(\text{name} + \text{"-"} + \text{resourceId})[0..11]$$

Testing internal DNS resolution from `client-terminal` to `test-redis`:

```bash
# DNS Resolution Test via nslookup
nslookup test-redis-feccbaec5bd7
```
**Actual Output:**
```text
Server:     10.43.0.10
Address:    10.43.0.10:53

Name:       test-redis-feccbaec5bd7.d62f9886-fced-4cf2-98e4-5b62000d4f03.svc.cluster.local
Address:    10.42.5.97
```

```bash
# POSIX Resolver Test via getent
getent hosts test-redis-feccbaec5bd7
```
**Actual Output:**
```text
10.42.5.97  test-redis-feccbaec5bd7.d62f9886-fced-4cf2-98e4-5b62000d4f03.svc.cluster.local test-redis-feccbaec5bd7
```

> [!NOTE]
> Bare hostname queries (e.g. `nslookup test-redis`) return `NXDOMAIN` because CoreDNS registers services under their full unique name (`test-redis-feccbaec5bd7`). Applications must either use the unique identifier or Ghaymah must configure DNS CNAME/A aliases for the bare application name within the project scope.

### 3.3 Raw TCP Connections Over Private Network

A raw TCP Redis server was provisioned with `expose: false` (strictly private network reachability). Raw non-HTTP TCP sockets were established from `client-terminal` directly to the DNS hostname `test-redis-feccbaec5bd7:6379`.

#### Test 3.3.1: Raw TCP Port Probe
```bash
nc -zv -w 3 test-redis-feccbaec5bd7 6379
```
**Output:**
```text
test-redis-feccbaec5bd7 (10.42.5.97:6379) open
PORT 6379 IS OPEN!
```

#### Test 3.3.2: Raw TCP Handshake (Redis PING)
```bash
printf "*1\r\n\$4\r\nPING\r\n" | nc -w 3 test-redis-feccbaec5bd7 6379
```
**Output:**
```text
+PONG
```

#### Test 3.3.3: Data Mutation & Retrieval Over TCP Socket
```bash
printf "*3\r\n\$3\r\nSET\r\n\$7\r\nghaymah\r\n\$16\r\nINTERNAL_DNS_OK\r\n" | nc -w 3 test-redis-feccbaec5bd7 6379
```
**Output:**
```text
+OK
```

**Verdict:** **PASS**. Containers discover and communicate over raw TCP sockets strictly via internal DNS resolution over the private network.

---

## 4. Phase 4: Security & Penetration Testing

### 4.1 Cross-Project Isolation (Security Finding)

**Test Objective:** Determine whether applications deployed in separate projects under the same account are network-isolated.

- **Source Container:** `client-terminal` in project `dns-tcp-test` (ID: `3f55ceef-78ae-40c8-a41b-0505db8c59ee`)
- **Target Container:** `auth-deploy` in project `auth-deploy` (ID: `5bd7e471-6be2-40a8-aff9-268beb29c631`)

#### Execution & Evidence:

1. **DNS Resolution of Cross-Project Service:**
   ```bash
   nslookup auth-deploy-4c0bca85312b
   ```
   **Output:**
   ```text
   Server:   10.43.0.10
   Name:     auth-deploy-4c0bca85312b.d62f9886-fced-4cf2-98e4-5b62000d4f03.svc.cluster.local
   Address:  10.42.4.223
   ```
   *Result:* Cross-project internal DNS resolved successfully.

2. **TCP Connectivity to Cross-Project Service:**
   ```bash
   nc -zv -w 3 auth-deploy-4c0bca85312b 80
   ```
   **Output:**
   ```text
   auth-deploy-4c0bca85312b (10.42.4.223:80) open
   ```
   *Result:* TCP port 80 is fully open and accessible across projects.

3. **HTTP Data Access Across Projects:**
   ```bash
   wget -qO- -T 3 http://auth-deploy-4c0bca85312b/
   ```
   **Output:**
   ```html
   <h1>test</h1>
   ```
   *Result:* The payload was retrieved directly over the internal network without public ingress traversal.

> [!WARNING]
> **Vulnerability Classification: Intra-Tenant Cross-Project Lateral Movement**
> **Severity:** **MEDIUM-HIGH**  
> While Ghaymah provides multi-tenant isolation (preventing one user from reaching another user's namespace), projects within the **same** user account share a single flat Kubernetes namespace (`d62f9886-fced-4cf2-98e4-5b62000d4f03`). No Kubernetes `NetworkPolicy` restricts traffic between different projects. A compromise in a public-facing development project allows lateral movement to internal databases or backend services in a production project under the same account.

### 4.2 Tenant & Cluster Infrastructure Isolation

Probes targeting external tenant environments and Kubernetes cluster infrastructure:

| Target Query / IP | Target Classification | Result | Verdict |
| :--- | :--- | :---: | :---: |
| `admin-panel` | Cross-Tenant Workload | `NXDOMAIN` | **BLOCKED (SECURE)** |
| `database.default` | Default Namespace Service | `NXDOMAIN` | **BLOCKED (SECURE)** |
| `kubernetes.default.svc` | Kubernetes API Service | `NXDOMAIN` | **BLOCKED (SECURE)** |
| `metadata.google.internal` | Cloud Metadata Service | `NXDOMAIN` | **BLOCKED (SECURE)** |
| `10.43.0.1:443` | Kubernetes API ClusterIP | Connection Timed Out | **BLOCKED (SECURE)** |
| `169.254.169.254:80` | Link-Local Cloud Metadata | Connection Timed Out | **BLOCKED (SECURE)** |
| `172.17.0.1:80` | Host Docker Bridge | Connection Timed Out | **BLOCKED (SECURE)** |
| `10.100.0.1` | WireGuard Gateway Probe | Connection Timed Out | **BLOCKED (SECURE)** |

**Verdict:** **PASS**. Cross-tenant leakage and cluster infrastructure access are completely mitigated.

### 4.3 Network Enumeration

Port scan executed from `client-terminal` against `test-redis-feccbaec5bd7`:

| Port | Service | Status | Notes |
| :---: | :---: | :---: | :--- |
| `22` | SSH | Closed | No SSH exposed on redis container |
| `53` | DNS | Closed | Not a DNS server |
| `80` | HTTP | Closed | No web server running |
| `443` | HTTPS | Closed | No TLS web server |
| `3000` | Node/Dev | Closed | Closed |
| `4000` | Backend | Closed | Closed |
| `5432` | PostgreSQL | Closed | Closed |
| `6379` | **Redis** | **OPEN** | **Active Redis server listening** |
| `8080` | HTTP Alt | Closed | Closed |
| `9000` | TCP Alt | Closed | Closed |

**Verdict:** **PASS**. Only the explicitly configured port (`6379`) is exposed.

### 4.4 Dual Routing & DNS Leak Detection

1. **Outbound Internet Egress:**
   - Command: `wget -qO- https://httpbin.org/ip`
   - Response: `{"origin": "184.107.141.203"}`
   - Egress Node: Leaseweb Canada Host Node (`cs221.mtl2.ca.leaseweb.net`). Outbound internet traffic is routed via source NAT as expected.

2. **External DNS Resolution:**
   - `google.com` $\rightarrow$ Resolved (`64.233.178.138`, `2607:f8b0:4020:c07::66`)
   - `github.com` $\rightarrow$ Resolved (`140.82.112.3`)
   - `cloudflare.com` $\rightarrow$ Resolved (`104.16.133.229`)
   - All external queries resolved through CoreDNS (`10.43.0.10`).

3. **DNS Leak Detection on Internal Domains:**
   - Query: `nslookup leak-test-nonexistent-internal-ghaymah.mesh`
   - Response: Strict `NXDOMAIN` from `10.43.0.10:53`. CoreDNS does not forward or leak `.mesh` or cluster queries upstream to public resolvers.

---

## 5. Security Recommendations & Hardening

1. **Implement Project-Level NetworkPolicies:**
   - Apply Kubernetes `NetworkPolicy` objects based on `project_id` labels:
     ```yaml
     apiVersion: networking.k8s.io/v1
     kind: NetworkPolicy
     metadata:
       name: project-isolation-dns-tcp-test
       namespace: d62f9886-fced-4cf2-98e4-5b62000d4f03
     spec:
       podSelector:
         matchLabels:
           ghaymah.systems/project-id: "3f55ceef-78ae-40c8-a41b-0505db8c59ee"
       ingress:
       - from:
         - podSelector:
             matchLabels:
               ghaymah.systems/project-id: "3f55ceef-78ae-40c8-a41b-0505db8c59ee"
     ```
   - This ensures containers in Project A cannot communicate with Project B unless an explicit peering or service link is enabled.

2. **CoreDNS Short-Name Aliasing:**
   - Currently, resolution requires the full unique name `test-redis-feccbaec5bd7`.
   - Implement CoreDNS rewrite or split-horizon plugin to allow short name resolution (`test-redis`) scoped to the pod's project tag.

3. **Restricted Egress Policies:**
   - Allow users to toggle "Internal-Only / No Egress" on sensitive backend databases to prevent data exfiltration if a container is compromised.

---

## 6. Verification Artifacts & Logs

All raw logs and test scripts are preserved in the workspace:
- **Terminal Execution Log:** [`ghaymah_execution.log`](file:///d:/cumin/ghaymah_dns_net_sec_tests/logs/ghaymah_execution.log)
- **Targeted UniqueName Log:** [`uniquename_dns_tcp.log`](file:///d:/cumin/ghaymah_dns_net_sec_tests/logs/uniquename_dns_tcp.log)
- **Automated Test Results JSON:** [`ghaymah_test_results.json`](file:///d:/cumin/ghaymah_dns_net_sec_tests/logs/ghaymah_test_results.json)
- **Automated Test Script:** [`test_uniquename_dns_tcp.js`](file:///d:/cumin/ghaymah_dns_net_sec_tests/scripts/test_uniquename_dns_tcp.js)