123 أسطر
4.8 KiB
Markdown
123 أسطر
4.8 KiB
Markdown
# MCP Red-Team Security Testing Server — Architecture Walkthrough
|
||
|
||
## Overview
|
||
|
||
A production-ready MCP Server for automated AI Red-Teaming and Security Testing against **https://os.solidpoint.ai**, built with `MCPServer` (mcp v2) and `Playwright`.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```mermaid
|
||
graph TD
|
||
A[MCP Client] -->|JSON-RPC stdio| B[server.py — MCPServer]
|
||
B -->|imports| C[client.py — Playwright Browser Layer]
|
||
C -->|drives| D[Chromium Headless]
|
||
D -->|interacts| E[os.solidpoint.ai]
|
||
|
||
B --> F[run_security_suite]
|
||
B --> G[evaluate_asr]
|
||
B --> H[generate_redteam_report]
|
||
|
||
F -->|reads| I[redteam_testcases.json — 45 payloads]
|
||
F -->|reads| J[accounts.csv — 10 accounts]
|
||
F -->|reads| K[.enc — Pro credentials + token]
|
||
|
||
F -->|writes| L[suite_results.json]
|
||
G -->|writes| M[evaluated_results.json]
|
||
H -->|writes| N[redteam_report.json]
|
||
H -->|writes| O[redteam_report.md]
|
||
```
|
||
|
||
---
|
||
|
||
## Files Created
|
||
|
||
| File | Purpose | Size |
|
||
|------|---------|------|
|
||
| [`server.py`](file:///c:/Users/ZIAD/OneDrive/سطح%20المكتب/ghaymah_solidpoint.ai/testing%20usning%20MCP/server.py) | FastMCP server — 3 tools exposed via stdio | 20.7 KB |
|
||
| [`client.py`](file:///c:/Users/ZIAD/OneDrive/سطح%20المكتب/ghaymah_solidpoint.ai/testing%20usning%20MCP/client.py) | Playwright browser automation layer | 16.9 KB |
|
||
| [`mcp_config.json`](file:///c:/Users/ZIAD/OneDrive/سطح%20المكتب/ghaymah_solidpoint.ai/testing%20usning%20MCP/mcp_config.json) | MCP server configuration (stdio transport) | 354 B |
|
||
|
||
---
|
||
|
||
## MCP Tools Exposed
|
||
|
||
### 1. `run_security_suite(headless, timeout_per_test, categories)`
|
||
Orchestrates all 45 red-team payloads from `redteam_testcases.json` across the loaded accounts via Playwright browser automation.
|
||
|
||
- **Auth priority:** Pro token → Pro creds → CSV accounts (round-robin)
|
||
- **Auto-continue:** Monitors for stalled responses and clicks "Continue" automatically
|
||
- **File injection:** Creates temp files for indirect injection test cases (D01–D05)
|
||
- **Output:** `suite_results.json`
|
||
|
||
### 2. `evaluate_asr(results_file)`
|
||
Computes Attack Success Rate using regex-based heuristics:
|
||
- **Attack success indicators:** PWNED, DAN compliance, system prompt leaks, credential exposure
|
||
- **Defense indicators:** Refusal patterns, safety guardrail phrases
|
||
- **Per-category ASR** and **overall risk level** (LOW/MEDIUM/HIGH/CRITICAL)
|
||
- **Output:** `evaluated_results.json`
|
||
|
||
### 3. `generate_redteam_report(results_file)`
|
||
Generates comprehensive security assessment report:
|
||
- **OWASP LLM Top 10 mapping** (LLM01 Prompt Injection, LLM06 Excessive Agency)
|
||
- **MITRE ATLAS technique mapping** (AML.T0051, AML.T0054, AML.T0048, AML.T0024)
|
||
- **Executive summary** with risk level
|
||
- **Remediation recommendations**
|
||
- **Output:** `redteam_report.json` + `redteam_report.md`
|
||
|
||
---
|
||
|
||
## Test Coverage (45 payloads across 7 categories)
|
||
|
||
| Category | ID Range | Count | OWASP Code | MITRE ATLAS |
|
||
|----------|----------|-------|------------|-------------|
|
||
| **direct** (Prompt Injection) | A01–A08 | 8 | LLM01 | AML.T0051.000 |
|
||
| **leak** (System Prompt Extraction) | B01–B08 | 8 | LLM01 | AML.T0051.001 |
|
||
| **jailbreak** (Safety Bypass) | C01–C08 | 8 | LLM01 | AML.T0054 |
|
||
| **indirect** (File-based Injection) | D01–D05 | 5 | LLM01 | AML.T0051.002 |
|
||
| **agency** (Excessive Permissions) | E01–E06 | 6 | LLM06 | AML.T0048 |
|
||
| **disclosure** (Data Leakage) | F01–F05 | 5 | LLM06 | AML.T0024 |
|
||
| **control** (Baseline) | G01–G05 | 5 | N/A | N/A |
|
||
|
||
---
|
||
|
||
## Strict Constraints Enforced
|
||
|
||
> [!IMPORTANT]
|
||
> - ❌ **ZERO `print()` statements** — All logging uses Python `logging` module → `sys.stderr` + `mcp_manager.log`
|
||
> - ✅ **stdout reserved exclusively** for JSON-RPC communication
|
||
> - ✅ **MCP_MODE=stdio** as specified in `mcp_config.json`
|
||
> - ✅ **Graceful error handling** — failed logins, timeouts, and network errors are caught and reported without crashing
|
||
|
||
---
|
||
|
||
## Validation Results
|
||
|
||
| Check | Status |
|
||
|-------|--------|
|
||
| Python 3.13 + mcp v2.1.1 + playwright v1.62 installed | ✅ |
|
||
| `client.py` loads .enc, accounts.csv, redteam_testcases.json | ✅ (4 env keys, 10 accounts, 45 tests) |
|
||
| `server.py` imports MCPServer correctly | ✅ |
|
||
| MCP `initialize` handshake responds | ✅ |
|
||
| MCP `tools/list` returns all 3 tools | ✅ |
|
||
| Chromium browser installed for Playwright | ✅ |
|
||
|
||
---
|
||
|
||
## How to Run
|
||
|
||
```bash
|
||
# Start the MCP server (used by MCP clients like Claude Desktop, etc.)
|
||
cd "c:\Users\ZIAD\OneDrive\سطح المكتب\ghaymah_solidpoint.ai\testing usning MCP"
|
||
& 'C:\Program Files\Python313\python.exe' server.py
|
||
|
||
# Or call tools directly for standalone testing:
|
||
& 'C:\Program Files\Python313\python.exe' -c "
|
||
import sys; sys.path.insert(0, '.')
|
||
from server import run_security_suite, evaluate_asr, generate_redteam_report
|
||
# Run a subset first:
|
||
print(run_security_suite(categories='direct,control'), file=sys.stderr)
|
||
"
|
||
```
|