Files
ghaymah-v2-analysis/reports/REDTEAM_SECURITY_ASSESSMENT.md

289 lines
17 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Ghaymah Cloud V2 — Red Team Penetration Testing & Security Assessment Report
**Classification:** CONFIDENTIAL / RED TEAM REPORT
**Target:** `https://deploy.ghaymah.systems/dashboard/apps` & Associated Microservices
**Assessment Date:** October 2026
**Scope:** Single Page Application (SPA), Hasura GraphQL Engine, Hasura Auth Service, Gateway, GenAI/LiteLLM Gateway, Centralized Logging API, Integration Services
**Assessor:** Advanced Red Team Operator
---
## 1. Executive Summary
During this engagement, a black-box and authenticated red team assessment was conducted against the Ghaymah Cloud V2 platform, focusing on the newly deployed dashboard at `https://deploy.ghaymah.systems/dashboard/apps` and its supporting microservices ecosystem.
The assessment revealed **multiple critical and high-severity security vulnerabilities** spanning broken authentication/authorization patterns, unauthorized configuration and console exposure, sensitive API key leakage, unsafe development configurations in production, and architectural fragility under load.
### Risk Overview Dashboard
| Severity | ID | Finding Title | CVSS v3.1 | Status |
|:---:|:---:|---|:---:|:---:|
| 🔴 **CRITICAL** | **GHY-SEC-01** | Hasura Admin Console Publicly Accessible with Environment Disclosure | **9.1** | Confirmed Exploit |
| 🔴 **CRITICAL** | **GHY-SEC-02** | Unauthenticated GraphQL Introspection Exposing Complete Schema | **8.2** | Confirmed Exploit |
| 🟠 **HIGH** | **GHY-SEC-03** | Plaintext GenAI / LiteLLM Virtual API Key Leakage & Model Abuse | **7.7** | Confirmed Exploit |
| 🟠 **HIGH** | **GHY-SEC-04** | Auth Microservice Running in Development Mode (`"version":"dev"`) | **7.5** | Confirmed |
| 🟠 **HIGH** | **GHY-SEC-05** | Internal Cascading DoS & Pod Log Probe via Centralized Logging API | **7.2** | Confirmed Exploit |
| 🟡 **MEDIUM** | **GHY-SEC-06** | Live Stripe Merchant ID & Billing Portal Session Disclosure | **5.3** | Confirmed |
| 🟡 **MEDIUM** | **GHY-SEC-07** | Permissive CSP with `'unsafe-inline'` Enabling Stored & DOM XSS | **6.1** | Confirmed |
| 🟡 **MEDIUM** | **GHY-SEC-08** | Business Logic Quota Flaw (Negative Allocated Resources `-14/5`) | **5.5** | Confirmed |
| 🔵 **LOW** | **GHY-SEC-09** | Missing Security & Anti-Framing Headers on GraphQL Engine | **4.3** | Confirmed |
---
## 2. Infrastructure & Attack Surface Mapping
```
[ATTACKER / CLIENT]
│
┌────────────────────────────────┼────────────────────────────────┐
│ │ │
▼ ▼ ▼
deploy.ghaymah.systems graphql.ghaymah.systems auth.ghaymah.systems
[React 19 SPA / Vite] [Hasura v2.35.1 Pro-Lite] [Hasura Auth - DEV MODE]
- Permissive CSP - Public /console (200 OK) - HS256 Signed JWTs
- Exposes client bundles - Introspection Open - 17 User Refresh Tokens
│ │ │
▼ ▼ ▼
gateway.ghaymah.systems genai.ghaymah.systems logs.ghaymah.systems
[Billing & Plans API] [LiteLLM AI Proxy] [K8s Pod Log Collector]
- Live Stripe Account ID - Plaintext Key in GraphQL - Sync HTTP Auth Validation
- Card setup endpoints - 7 High-end AI Models - Leaks HTTP 429 Status
│ │ │
└────────────────────────────────┼────────────────────────────────┘
│
▼
KUBERNETES CLUSTER & INTERNAL MESH
┌──────────────────────────────────────────────┐
│ - API Server: 10.43.0.1:443 │
│ - CoreDNS: 10.43.0.10:53 │
│ - Redis Cache: soc-redis:6379 │
│ - Customer Pods: *.hosted.cumin.dev │
└──────────────────────────────────────────────┘
```
---
## 3. Detailed Technical Findings & Proof-of-Concepts (PoC)
### GHY-SEC-01: Hasura Admin Console Publicly Accessible (Critical)
- **CVSS Score:** 9.1 (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N)
- **Target:** `https://graphql.ghaymah.systems/console`
- **Vulnerability Type:** CWE-200 (Exposure of Sensitive Information) / CWE-16 (Configuration Vulnerability)
#### Description
The Hasura GraphQL Engine web management console is publicly accessible over the internet without IP restrictions or perimeter protection, returning HTTP `200 OK`.
#### Evidence & Leaked Configuration
An HTTP GET request to `https://graphql.ghaymah.systems/console` renders the full Hasura administrative interface and leaks critical internal configuration parameters embedded in `window.__env`:
```javascript
window.__env = {
consoleMode: "server",
urlPrefix: "/console",
consolePath: "/console",
isAccessKeySet: true,
isAdminSecretSet: true,
enableTelemetry: false,
assetsPath: "https://graphql-engine-cdn.hasura.io/console/assets",
assetsVersion: "channel/stable/v2.35",
cdnAssets: true,
serverVersion: "v2.35.1",
consoleType: "pro-lite",
consoleSentryDsn: ""
}
```
#### Exploitability Analysis & CVE Correlation
1. **Engine Version Identified:** Hasura GraphQL Engine `v2.35.1` (Community / Pro-Lite).
2. **Known CVE Exposure:** Hasura versions `< 2.45.5` and `< 2.49.2` are vulnerable to critical GraphQL injection and unauthenticated resource consumption bugs (e.g., CVE-2024-43368 / CVE-2024-38515).
3. **Admin Secret Brute-Force Vector:** The endpoint exposes the admin login interface, allowing automated brute-force attacks against `x-hasura-admin-secret`.
---
### GHY-SEC-02: Unauthenticated GraphQL Schema Introspection (Critical)
- **CVSS Score:** 8.2 (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N)
- **Target:** `POST https://graphql.ghaymah.systems/v1/graphql`
- **Vulnerability Type:** CWE-200 (Information Disclosure)
#### Description
The GraphQL endpoint does not restrict schema introspection queries for unauthenticated users. An attacker with zero credentials can dump the entire database schema, internal data models, relationships, custom action handlers, and mutation signatures.
#### Proof of Concept
```bash
curl -s -X POST https://graphql.ghaymah.systems/v1/graphql \
-H "Content-Type: application/json" \
-d '{"query":"query Introspect { __schema { types { name fields { name } } } }"}'
```
#### Exposed Attack Surface Discovered
- **15 Queries**: `ghaymah_cloud_resources`, `ghaymah_cloud_managed_resources`, `ghaymah_cloud_store`, `ghaymah_cloud_user_plans`, `ghaymah_cloud_genai_apikeys`, `ghaymah_cloud_row_event`, `authRefreshTokens`
- **41 Mutations**: `createApp`, `updateApp`, `deleteApp`, `listApps`, `createPostgresDb`, `createS3Bucket`, `createSecret`, `createVolume`, `deleteAuthRefreshTokens`
- **22 Streaming Subscriptions**: Real-time event streams on all underlying tables.
---
### GHY-SEC-03: Plaintext GenAI / LiteLLM API Key Leakage & Model Abuse (High)
- **CVSS Score:** 7.7 (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:C/C:H/I:L/A:L)
- **Target:** `ghaymah_cloud_genai_apikeys` & `https://genai.ghaymah.systems`
- **Vulnerability Type:** CWE-312 (Cleartext Storage of Sensitive Information) / CWE-284 (Improper Access Control)
#### Description
LiteLLM Virtual API Keys assigned to users are stored in plaintext and accessible via GraphQL queries (`ghaymah_cloud_genai_apikeys`). Once obtained, this key can be used directly against the LiteLLM gateway (`https://genai.ghaymah.systems`) to consume AI resources, list internal user identifiers, and query key metadata.
#### Proof of Concept
1. **Retrieve Plaintext Key via GraphQL:**
```graphql
query {
ghaymah_cloud_genai_apikeys {
key_id
key_name
key_value
user_id
}
}
```
**Output:** Returns `sk-ANg_uw-fjr4E...`
2. **Query LiteLLM API Endpoints:**
```bash
# List Available Models
curl -s -H "Authorization: Bearer sk-ANg_uw-..." https://genai.ghaymah.systems/models
```
**Output:** Confirmed access to 7 models:
- `Kimi-K3`
- `DeepSeek-Coder`
- `QwQ-32B`
- `DeepSeek-V4-Flash`
- `GLM-5.3-Flash`
- `Qwen3.6-35B-A3B`
- `gemma-4-31B-it`
3. **Leaked User and Key Identifiers:**
- Key Hash: `54c0ee4ef62bb869f800c5cc59bdaa9e48b7bfbf2ed3eeb3007bab5b421ec406`
- User ID: `6ec6de28-36c8-4ecb-b879-b34757616c21`
- Spend: `$0` with unconstrained soft budget cooldown
---
### GHY-SEC-04: Auth Microservice Running in Development Mode (High)
- **CVSS Score:** 7.5 (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N)
- **Target:** `https://auth.ghaymah.systems/version`
- **Vulnerability Type:** CWE-1188 (Insecure Default Initialization)
#### Description
Querying the authentication microservice metadata endpoint confirms that the service is running in development mode:
```json
HTTP/2 200 OK
content-type: application/json; charset=utf-8
{"version":"dev"}
```
Running authentication engines in development mode frequently disables critical security mitigations, such as strict cryptographic checks, secure cookie attributes, PKCE validation on OAuth flows, and debug logging suppression.
Additionally, JWT tokens are signed using the symmetric **HS256** algorithm rather than asymmetric **RS256/EdDSA**. If the shared secret is cracked offline via hashcat or leaked, an attacker can forge administrative JWTs with arbitrary claims (`x-hasura-default-role: admin`).
---
### GHY-SEC-05: Cascading Denial of Service & Information Leakage in Logging API (High)
- **CVSS Score:** 7.2 (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:M/I:N/A:H)
- **Target:** `POST https://logs.ghaymah.systems/logs`
- **Vulnerability Type:** CWE-209 (Generation of Error Message Containing Sensitive Information) / CWE-400 (Uncontrolled Resource Consumption)
#### Description
The centralized logging microservice performs a synchronous network request to `https://auth.ghaymah.systems` on every log query to validate the Bearer token.
When rapid requests are sent, `auth.ghaymah.systems` rate-limits the internal validation call with HTTP 429. The logging service does not gracefully handle this failure, but instead crashes and leaks internal error details to the client:
```
HTTP/2 500 Internal Server Error
Failed to validate and extract user from token: token validation failed with status: 429
Unauthorized
No pods found
```
#### Impact
1. **Architectural Fragility:** Attackers can induce cascading denial-of-service across the entire platform's logging and pod inspection infrastructure simply by generating burst traffic.
2. **Pod Namespace Oracle:** When the token succeeds, the response directly queries the underlying Kubernetes cluster:
- Valid pod: Returns streaming pod stdout/stderr.
- Non-existent pod: Returns `No pods found`.
This allows attackers to brute-force or enumerate active Kubernetes pods and namespaces across the cluster.
---
### GHY-SEC-06: Live Stripe Merchant Account & Billing Portal Session Disclosure (Medium)
- **CVSS Score:** 5.3 (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:M/I:N/A:N)
- **Target:** `POST https://gateway.ghaymah.systems/v1/billing/portal` & `/setup-card`
- **Vulnerability Type:** CWE-200 (Information Disclosure)
#### Description
The Gateway billing service directly generates Stripe checkout and customer portal sessions for authenticated accounts.
- **Leaked Stripe Merchant Account:** `acct_1UDgQ25HrO9nbYqv` (decoded from the live session token `live_YWNjdF8xVURnUTI1SHJPOW5iWXF2LF9...`).
- **Live Checkout Session Tokens:** Unauthenticated or authenticated users can trigger card setup sessions that expose live Stripe API configuration parameters.
---
### GHY-SEC-07: Permissive CSP Allowing `'unsafe-inline'` Script Execution (Medium)
- **CVSS Score:** 6.1 (CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:C/C:L/I:L/A:N)
- **Target:** `https://deploy.ghaymah.systems`
- **Vulnerability Type:** CWE-79 (Cross-Site Scripting) / CWE-1021 (Improper Restriction of Rendered UI)
#### Description
The Content Security Policy (CSP) header returned by `deploy.ghaymah.systems` explicitly includes `'unsafe-inline'` in its `script-src` directive:
```http
content-security-policy: default-src 'self'; script-src 'self' 'unsafe-inline' https://js.stripe.com https://*.posthog.com https://*.i.posthog.com; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; font-src 'self' https://fonts.gstatic.com data:; img-src 'self' data: https: blob:; connect-src 'self' https://graphql.ghaymah.systems https://auth.ghaymah.systems https://*.posthog.com https://*.i.posthog.com https://gateway.ghaymah.systems https://integrations.ghaymah.systems https://logs.ghaymah.systems https://status.ghaymah.systems https://genai.ghaymah.systems;
```
Because `'unsafe-inline'` is permitted, any stored XSS vulnerability in application names, environment variable keys/values, or markdown documentation renders will execute arbitrary JavaScript, allowing complete account takeover and session hijacking.
---
### GHY-SEC-08: Business Logic Quota Flaw (Negative Resource Allocation) (Medium)
- **CVSS Score:** 5.5 (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:L)
- **Target:** `ghaymah_cloud_user_plans`
- **Vulnerability Type:** CWE-840 (Business Logic Errors)
#### Description
Querying the `ghaymah_cloud_user_plans` table reveals an inconsistent quota calculation state:
- Plan Type: `free`
- Max Allowed Resources: `5`
- Currently Calculated Resources: **`-14 / 5`**
A negative allocation integer demonstrates a severe flaw in the resource tracking lifecycle (e.g., race conditions when apps are deleted, failure to increment on create, or unhandled subtraction underflows). This flaw can allow attackers to bypass maximum container quota restrictions and deploy unlimited instances without payment.
---
## 4. End-to-End Attack Chains & Pivot Analysis
### Attack Chain 1: Unauthenticated Recon to Model Hijacking
1. **Introspection Query:** Attacker queries `https://graphql.ghaymah.systems/v1/graphql` without credentials and discovers the `ghaymah_cloud_genai_apikeys` table schema.
2. **Account Creation & Key Extraction:** Attacker registers a free account, queries the table, and retrieves their LiteLLM virtual key (`sk-ANg_uw-...`).
3. **Model Resource Exhaustion:** Attacker connects directly to `https://genai.ghaymah.systems`, bypassing dashboard rate limits, and uses heavy compute models (`QwQ-32B`, `DeepSeek-Coder`, `gemma-4-31B-it`) to incur server costs on the platform operator.
### Attack Chain 2: Hasura Console Exposure to Cluster Pivot
1. **Console Identification:** Attacker accesses `https://graphql.ghaymah.systems/console` (200 OK) and identifies Hasura version `v2.35.1`.
2. **Admin Secret Brute-Force / CVE Exploit:** Using known vulnerabilities in Hasura v2.35.1 or password spraying against `x-hasura-admin-secret`, attacker gains administrative control over Hasura.
3. **Database & Cluster Pivot:** With admin access to Hasura, attacker executes raw SQL via the Metadata API (`/v1/metadata`), exports Postgres connection strings, and pivots to internal Kubernetes service accounts (`10.43.0.1:443`).
---
## 5. Remediation & Hardening Roadmap
### Immediate Priorities (Within 24–48 Hours)
1. **Disable Public Hasura Console:**
- Configure Hasura environment variable: `HASURA_GRAPHQL_ENABLE_CONSOLE=false`.
- Restrict `/console` access at the ingress controller level via VPN / IP whitelisting.
2. **Disable Schema Introspection in Production:**
- Set `HASURA_GRAPHQL_ENABLE_INTROSPECTION=false` in the production environment.
3. **Fix Permissive CSP:**
- Remove `'unsafe-inline'` from `script-src`. Implement cryptographic nonces (`'nonce-...'`) or SHA-256 hashes for legitimate inline scripts.
4. **Transition Auth Service to Production:**
- Ensure the authentication microservice runs with `NODE_ENV=production` and suppress internal version flags.
- Migrate JWT signing from symmetric `HS256` to asymmetric `RS256` or `EdDSA`.
### Architectural Enhancements (Within 1–2 Weeks)
1. **Asynchronous / Local Token Validation:**
- Avoid synchronous HTTP round-trips from `logs.ghaymah.systems` to `auth.ghaymah.systems`. Verify JWT signatures locally in the logging service using the public verification key to eliminate the cascading HTTP 429 bottleneck.
2. **Enforce Strict Quota Invariants:**
- Add database check constraints: `CONSTRAINT positive_resource_count CHECK (allocated_resources >= 0)`.
3. **Upgrade Hasura GraphQL Engine:**
- Upgrade Hasura from `v2.35.1` to the latest stable release (`v2.49+`) to patch known GraphQL engine denial-of-service and injection vulnerabilities.