162 أسطر
6.0 KiB
Markdown
162 أسطر
6.0 KiB
Markdown
<div align="center">
|
|
|
|
# 🛡️ MCP Red-Team Security Server
|
|
|
|
**Automated AI Security & Penetration Testing Framework**
|
|
|
|
[](https://www.python.org/)
|
|
[](https://modelcontextprotocol.io/)
|
|
[](https://playwright.dev/)
|
|
[](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
|
|
[](https://atlas.mitre.org/)
|
|
|
|
</div>
|
|
|
|
---
|
|
|
|
## 📖 Overview
|
|
|
|
The **MCP Red-Team Security Server** is a production-grade automated security testing suite built on the **Model Context Protocol (MCP)**. It uses **Playwright** to drive browser-based injection payloads against an LLM platform (target: `os.solidpoint.ai`).
|
|
|
|
It programmatically injects 45 crafted adversarial payloads, monitors the model's responses, heuristically evaluates the **Attack Success Rate (ASR)**, and generates a final security report mapped directly to the **OWASP LLM Top 10** and **MITRE ATLAS** frameworks.
|
|
|
|
---
|
|
|
|
## 🏗️ Architecture & Workflow
|
|
|
|
The framework operates in a completely headless and automated manner, coordinating an MCP client, the MCP Server, and the browser automation layer.
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant User/MCP Client
|
|
participant Server as MCP Server (server.py)
|
|
participant Client as Playwright Layer (client.py)
|
|
participant Target as Target LLM Platform
|
|
|
|
User/MCP Client->>Server: Call `run_security_suite()`
|
|
Server->>Client: Load 45 payloads & Accounts
|
|
|
|
loop For each payload
|
|
Client->>Target: Headless Login (Token/Creds)
|
|
Client->>Target: Inject Payload (Chat / File Upload)
|
|
Target-->>Client: AI Response (Streaming)
|
|
Client->>Client: Monitor & Extract Final Response
|
|
end
|
|
|
|
Client-->>Server: Raw Suite Results
|
|
|
|
User/MCP Client->>Server: Call `evaluate_asr()`
|
|
Server->>Server: Regex Heuristics & Analysis
|
|
Server-->>User/MCP Client: Evaluated Results (ASR)
|
|
|
|
User/MCP Client->>Server: Call `generate_redteam_report()`
|
|
Server->>Server: Map to OWASP & MITRE ATLAS
|
|
Server-->>User/MCP Client: Markdown & JSON Reports
|
|
```
|
|
|
|
---
|
|
|
|
## 📂 Project Structure
|
|
|
|
```text
|
|
RedTeam-MCP-Server/
|
|
├── src/ # Source Code
|
|
│ ├── server.py # FastMCP Server (MCP v2) exposing the 3 tools
|
|
│ ├── client.py # Playwright browser automation layer
|
|
│ └── run_all.py # Standalone orchestrator script
|
|
├── data/ # Inputs
|
|
│ ├── redteam_testcases.json # 45 Injection Payloads
|
|
│ ├── accounts.csv # Test Accounts
|
|
│ └── created_accounts.json
|
|
├── config/ # Configuration
|
|
│ ├── mcp_config.json # MCP Client Configuration (stdio)
|
|
│ └── .env # Target URL, Pro Credentials & Tokens
|
|
├── reports/ # Outputs
|
|
│ ├── suite_results.json
|
|
│ ├── evaluated_results.json
|
|
│ ├── redteam_report.json
|
|
│ └── redteam_report.md
|
|
└── docs/ # Security Guides
|
|
├── LLM_Testing_AI_Security_Pentesting_Prompt_Injection_Guide.md
|
|
└── LLM_Testing_Security_Pentesting_PromptInjection_Guide.md
|
|
```
|
|
|
|
---
|
|
|
|
## 🛠️ Exposed MCP Tools
|
|
|
|
The server (`src/server.py`) exposes three tools via `stdio` transport:
|
|
|
|
1. `run_security_suite(headless=True, timeout_per_test=120, categories="")`
|
|
* Opens the browser, handles authentication, and executes all defined testcases.
|
|
2. `evaluate_asr(results_file="")`
|
|
* Processes the raw responses to calculate the **Attack Success Rate (ASR)**.
|
|
* Classifies responses as `ATTACK_SUCCEEDED`, `DEFENDED`, `POSSIBLE_LEAK`, etc.
|
|
3. `generate_redteam_report(results_file="")`
|
|
* Compiles the evaluated data into a comprehensive Markdown report mapped to **OWASP LLM01/LLM06** and **MITRE ATLAS AML.T0051/AML.T0048**.
|
|
|
|
---
|
|
|
|
## 🚀 Getting Started
|
|
|
|
### Prerequisites
|
|
- Python 3.13+
|
|
- Required packages: `mcp[cli]`, `playwright`
|
|
|
|
```bash
|
|
# Install dependencies
|
|
pip install "mcp[cli]" playwright
|
|
playwright install chromium
|
|
```
|
|
|
|
### Running Standalone (No MCP Client Required)
|
|
To run the full suite automatically without an MCP Client:
|
|
|
|
```bash
|
|
cd RedTeam-MCP-Server
|
|
python src/run_all.py
|
|
```
|
|
|
|
### Running via MCP
|
|
Add the server to your MCP Client configuration (e.g., Claude Desktop, Cursor, or IDE Extensions) using the provided `config/mcp_config.json`:
|
|
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"redteam-security": {
|
|
"command": "C:\\Program Files\\Python313\\python.exe",
|
|
"args": ["src\\server.py"],
|
|
"cwd": "C:\\path\\to\\RedTeam-MCP-Server",
|
|
"env": {
|
|
"MCP_MODE": "stdio",
|
|
"PYTHONUNBUFFERED": "1"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 Evaluation Categories (45 Payloads)
|
|
|
|
| Category | Description | OWASP | MITRE ATLAS |
|
|
|----------|-------------|-------|-------------|
|
|
| **Direct** | Direct Prompt Injections (e.g., "Ignore instructions") | LLM01 | AML.T0051.000 |
|
|
| **Leak** | System Prompt Extraction & Reconnaissance | LLM01 | AML.T0051.001 |
|
|
| **Jailbreak**| Safety filter bypass (e.g., DAN, Roleplay) | LLM01 | AML.T0054 |
|
|
| **Indirect** | Injection via file uploads (e.g., poisoned `.txt` / `.csv`) | LLM01 | AML.T0051.002 |
|
|
| **Agency** | Exploiting system tools (e.g., executing commands) | LLM06 | AML.T0048 |
|
|
| **Disclosure**| Data leakage of private environments/keys | LLM06 | AML.T0024 |
|
|
| **Control** | Baseline generic questions to ensure model stability | N/A | N/A |
|
|
|
|
---
|
|
|
|
## 🔒 Security Constraints Enforced
|
|
|
|
- **Zero `print()` Statements**: All logging uses the Python `logging` module to output to `sys.stderr` and `logs/mcp_manager.log` to preserve JSON-RPC integrity on `stdout`.
|
|
- **Fault Tolerance**: Stalled generation triggers automatic UI 'Continue' clicks. Failed logins skip gracefully to the next test.
|
|
|
|
---
|
|
*Developed for internal AI Red-Teaming and Security Assessment.*
|