Ghyamah-task-1-cli2/Readme.md

# Ghaymah CLI — Manual Test Report

> **CLI Version:** `gy v0.0.24`  
> **Test Date:** 2026-08-21  
> **Tester:** Mahmoud

---

## Table of Contents

1. [Overview](#overview)
2. [Test Summary](#test-summary)
3. [Core Commands](#core-commands)
   - [version](#1-version)
   - [help](#2-help)
   - [signup](#3-signup)
   - [login](#4-login)
   - [logout](#5-logout)
   - [whoami](#6-whoami)
   - [list](#7-list)
4. [Deployment Tests](#deployment-tests)
   - [Node.js](#nodejs-deployment)
   - [Flask](#flask-deployment)
   - [Go (Non-Web)](#go-non-web-deployment)
   - [Go (Web)](#go-web-deployment)
   - [FastAPI](#fastapi-deployment)
5. [Tunneling](#tunneling)
6. [Delete Operations](#delete-operations)
7. [Security Tests](#security-tests)
   - [Secret Exposure in CLI Output](#secret-exposure-in-cli-output)
   - [Sensitive File Inclusion](#sensitive-file-inclusion)
   - [Git Metadata Exposure](#git-metadata-exposure)
   - [Dockerfile Hardcoded Secret Detection](#dockerfile-hardcoded-secret-detection)
   - [AI-Assisted Secret Detection](#ai-assisted-secret-detection)
8. [Known Issues & Bugs](#known-issues--bugs)
9. [Verdict Summary Table](#verdict-summary-table)

---

## Overview

This document reports the results of manual functional and security testing performed on the **Ghaymah CLI (`gy`)**. Tests cover core authentication commands, multi-language deployment scenarios, tunneling, resource management, and several security-focused scenarios including secret exposure and file inclusion.

---

## Test Summary

| Category | Total Tests | ✅ Pass | ⚠️ Partial | ❌ Fail |
|---|---|---|---|---|
| Core Commands | 7 | 6 | 0 | 1 |
| Deployments | 5 | 3 | 1 | 1 |
| Tunneling | 1 | 0 | 0 | 1 |
| Delete Operations | 2 | 1 | 1 | 0 |
| Security Tests | 5 | 2 | 0 | 3 |
| **Total** | **20** | **12** | **2** | **5** |

---

## Core Commands

### 1. Version

| Field | Details |
|---|---|
| **Command** | `gy version` |
| **Expected** | Display current CLI version and update status |
| **Result** | ✅ Pass |

**Output:**
```
Ghaymah CLI v0.0.24
You're up to date!
```

The CLI correctly reports its current version and confirms it is up to date.

---

### 2. Help

| Field | Details |
|---|---|
| **Command** | `gy --help` |
| **Expected** | Display CLI description, available commands, usage examples, and global flags |
| **Result** | ✅ Pass |

The help output is clear and provides all necessary information about the available commands and CLI options.

---

### 3. Signup

| Field | Details |
|---|---|
| **Command** | `gy signup` |
| **Expected** | Create a new account and auto-login |
| **Result** | ✅ Pass |

**Output:**
```
✅ Account created! Logged in as username@gmail.com
```

> **Note:** An informational log `config file not found, using defaults` appeared during signup. This is expected behavior and is not an issue.

---

### 4. Login

| Field | Details |
|---|---|
| **Command** | `gy login` |
| **Expected** | Authenticate with existing Ghaymah account credentials |
| **Result** | ✅ Pass |

Login succeeds using the email associated with the Ghaymah account. If no account exists, `gy signup` should be used instead.

---

### 5. Logout

| Field | Details |
|---|---|
| **Command** | `gy logout` |
| **Expected** | Sign out and clear local session |
| **Result** | ⚠️ Non-blocking issue |

**Output:**
```
Remote sign-out call failed with 400 error: "refreshToken" is required
✅ Logged out
```

Logout succeeds **locally** and clears the local config. However, the **remote sign-out request fails** because the `refreshToken` is not being included in the request body.

> **Bug:** The API call for remote sign-out is missing the required `refreshToken` field. The session may remain active server-side until token expiry.

---

### 6. Whoami

| Field | Details |
|---|---|
| **Command** | `gy whoami` |
| **Expected** | Display the currently authenticated user's email and User ID |
| **Result** | ✅ Pass |

**Output:**
```
✅ Logged in as username@gmail.com
User ID: <id>
```

No authentication tokens or secrets are exposed. The command correctly identifies the logged-in user.

---

### 7. List

| Field | Details |
|---|---|
| **Command** | `gy list` |
| **Expected** | List all projects associated with the authenticated account |
| **Result** | ✅ Pass |

**Output:**
```
[proj] synced ghaymah-exam-mahmoud-secops
```

Projects are listed with their synchronization status as expected.

---

## Deployment Tests

### Node.js Deployment

| Field | Details |
|---|---|
| **Command** | `gy deploy` |
| **Requirements** | `package.json`, `package-lock.json`, HTTP server listening on port `8080` |
| **Result** | ✅ Pass |

**Steps:**
1. Created a minimal Node.js Hello World project.
2. Initial attempt without a lockfile failed with an `npm ci` error → generated `package-lock.json`.
3. First runtime test failed because `index.js` was missing and later returned `502` because the app wasn't listening on port `8080`.
4. Updated `index.js` to start an HTTP server on port `8080`, redeployed, and verified using `gy config`, `gy logs`, and the deployed URL.

**Output:**
```
Hello from Node.js!
```

> **Note:** The project **must** include a `package-lock.json` and the application **must** listen on port `8080` (or the configured port) for deployment to succeed.

---

### Flask Deployment

| Field | Details |
|---|---|
| **Command** | `gy deploy --port=5000` |
| **Requirements** | Flask application listening on the configured port |
| **Result** | ✅ Pass |

**Steps:**
1. Initial deployment using the default port `8080` resulted in `502 Bad Gateway`.
2. Changed the application to listen on port `5000`.
3. Redeployed with `gy deploy --port=5000` — deployment completed successfully.

> **Note:** The `--port` flag must match the port the application actually listens on.

---

### Go (Non-Web) Deployment

| Field | Details |
|---|---|
| **Command** | `gy deploy` |
| **Requirements** | Valid Go project |
| **Result** | ❌ Fail |

**Steps:**
1. Deployed a simple Go program that prints `Hello from Go!` — not a web server.
2. The CLI generated a Dockerfile, built, and uploaded the artifact successfully.
3. However, resource creation failed with a `resources_pkey` uniqueness violation.
4. In an earlier attempt, the CLI waited for **15 minutes** before timing out, while `gy logs go-test` showed `Hello from Go!`.

**Output:**
```
resources_pkey uniqueness violation
(earlier) Deployment timeout after 15 minutes
Hello from Go!  ← from logs
```

> **Bug:** The CLI successfully detects and builds the Go application, but deployment does not complete and leaves the state inconsistent. See [Known Issues](#known-issues--bugs).

---

### Go (Web) Deployment

| Field | Details |
|---|---|
| **Command** | `gy deploy --port=8000` |
| **Requirements** | Go HTTP server listening on the configured port |
| **Result** | ✅ Pass |

**Steps:**
1. Created a minimal Go HTTP server using `net/http` listening on port `8000`.
2. Deployed without providing a Dockerfile — the CLI auto-detected Go and generated one.
3. Deployment completed and the app was reachable.

**Output:**
```
Hello from Go Web!
```

The CLI correctly detects, builds, and deploys a Go web application when the app listens on the configured port.

---

### FastAPI Deployment

| Field | Details |
|---|---|
| **Command** | `gy deploy --port=8000` |
| **Requirements** | FastAPI application with Uvicorn listening on `0.0.0.0:8000` |
| **Result** | ⚠️ Partial — CLI Issue |

**Steps:**
1. Deployed a FastAPI application using the CLI-generated Dockerfile.
2. Deployment completed successfully, but accessing the URL returned `502 Bad Gateway`.
3. Investigation revealed the auto-generated Dockerfile used `CMD ["sh", "-c", "python main.py"]`.
4. Since `main.py` only defines the FastAPI app object and does not start a server, the container never listens on the port.
5. Verified locally using `uvicorn main:app --host 0.0.0.0 --port 8000` — worked correctly.

**Output:**
```
Deployment completed successfully.
URL: 502 Bad Gateway
```

> **Bug:** The auto-generated Dockerfile for FastAPI uses `python main.py` instead of a Uvicorn command. The correct CMD should be:
> ```dockerfile
> CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
> ```

---

## Tunneling

| Field | Details |
|---|---|
| **Command** | `gy tunnel start <name>` |
| **Requirements** | A successfully deployed application with a configured port |
| **Result** | ❌ Fail |

**Steps:**
1. Started the tunnel for a deployed application.
2. The CLI reported the tunnel as live but the endpoint was unreachable.
3. The tunnel incorrectly targeted `localhost:8080` instead of the configured application port.
4. The tunnel URL returned was empty: `https://`.

**Output:**
```
Tunnel is live at: https://
```

> **Bug:** Three separate issues were identified (see [Known Issues](#known-issues--bugs)):
> - Hardcoded port `8080` instead of using the configured application port.
> - Empty tunnel URL returned.
> - Tunnel reported as live despite being unreachable.

---

## Delete Operations

### Delete Project

| Field | Details |
|---|---|
| **Command** | `gy delete project ghaymah-exam-mahmoud-secops` |
| **Result** | ✅ Pass |

**Output:**
```
✅ Deleted project 'ghaymah-exam-mahmoud-secops'
```

A confirmation prompt appeared before deletion. The project and all its associated apps were deleted successfully.

---

### Delete App

| Field | Details |
|---|---|
| **Command** | `gy delete app <name>` |
| **Result** | ⚠️ Works for unique names; ambiguous for duplicates |

App deletion works correctly when the app name is unique. However, if **duplicate app names exist**, the CLI cannot uniquely identify the target and the deletion becomes unreliable.

> **Bug:** The CLI relies solely on the friendly name for app deletion. Duplicate names should either be prevented or the CLI should support deletion by unique resource ID or interactive selection.

---

## Security Tests

### Secret Exposure in CLI Output

| Field | Details |
|---|---|
| **Command** | `gy deploy --debug` |
| **Test** | Checked whether fake credentials in `.env` and `config.txt` appear in CLI output or app logs |
| **Result** | ✅ Pass |

Deployment completed successfully. The CLI output and application logs did **not** contain the test credentials.

---

### Sensitive File Inclusion

| Field | Details |
|---|---|
| **Command** | `gy deploy --debug` |
| **Test** | Verified whether `.dockerignore` rules are honored and sensitive files are excluded |
| **Result** | ✅ Pass (with `.dockerignore`) / ❌ Fail (without `.dockerignore`) |

**With `.dockerignore`:**
Created `.dockerignore` rules excluding `.env` and `config.txt`. The deployed container contained only the allowed files.

```
total 16
.dockerignore
Dockerfile
```

**Without `.dockerignore`:**
Deployed a project containing `.env` and `config.txt` without a `.dockerignore`. Both files were present in the deployed container.

```
.env
Dockerfile
config.txt
```

> **Security Issue:** Sensitive files are **not automatically excluded** from the deployment. Developers must manually configure `.dockerignore`. The CLI should warn users when sensitive-looking files (e.g., `.env`, `*.key`, `config.txt`) are included in the build context.

---

### Git Metadata Exposure

| Field | Details |
|---|---|
| **Command** | `gy deploy --debug` |
| **Test** | Verified whether `.git` directory and metadata are included in the deployed container |
| **Result** | ✅ Pass |

Created a Git repository and committed project files. Added a Dockerfile that checks for `.git` and `.git/config`. The deployed container did **not** contain the `.git` directory.

**Output:**
```
=== GIT CHECK ===
ls: .git: No such file or directory
=== CONFIG CHECK ===
cat: can't open '.git/config': No such file or directory
```

---

### Dockerfile Hardcoded Secret Detection

| Field | Details |
|---|---|
| **Command** | `gy deploy --debug` |
| **Test** | Verified whether the CLI warns about hardcoded secrets in `ENV` instructions |
| **Result** | ❌ Fail |

Created a Dockerfile with `ENV TEST_SECRET="FAKE_SECRET_12345"`. Deployed the application and inspected the output and logs.

**Output:**
```
Application started
TEST_SECRET=FAKE_SECRET_12345
```

The CLI deployed the application **without any security warning**. The hardcoded secret was accessible inside the running container and appeared in the application logs.

> **Security Issue:** The CLI does not scan or warn about hardcoded secrets in `ENV` instructions within the Dockerfile.

---

### AI-Assisted Secret Detection

| Field | Details |
|---|---|
| **Command** | `gy deploy --debug --ai-key="$GY_AI_KEY"` |
| **Test** | Verified whether the `--ai-key` flag enables detection of hardcoded secrets in source code |
| **Result** | ❌ Fail |

Created a test project with a hardcoded API key in `app.sh`. Deployed with `--ai-key` flag enabled. No security warning or detection message was observed.

**Output:**
```
Application started
```

> **Security Issue:** The AI-assisted secret detection feature (`--ai-key`) did not identify or warn about the hardcoded API key. This feature appears to be non-functional or not yet implemented.

---

## Known Issues & Bugs

### 🐛 Bug-01 — Duplicate App Names Cause Ambiguity

- **Affected Command:** `gy delete app <name>`
- **Description:** The CLI identifies apps by friendly name only. When multiple apps share the same name, the CLI cannot uniquely target an app for deletion.
- **Recommendation:** Prevent duplicate app names at creation time, or support deletion by unique resource ID / interactive disambiguation.

---

### 🐛 Bug-02 — Default Port Assumption Causes 502 Errors

- **Affected Commands:** `gy deploy`, `gy tunnel start`
- **Description:** The CLI assumes port `8080` by default. Applications listening on a different port will result in `502 Bad Gateway`.
- **Recommendation:** Detect the actual application port automatically, or require explicit port configuration with a clearer error message.

---

### 🐛 Bug-03 — Tunnel Fails Completely

- **Affected Command:** `gy tunnel start <name>`
- **Description:** Three compounding issues:
  1. Port is hardcoded to `8080` instead of using the configured application port.
  2. The tunnel URL is returned empty (`https://`).
  3. The tunnel is reported as **live** despite being unreachable.
- **Recommendation:** Fix port synchronization with deployment config, validate the tunnel URL before reporting success, and surface a proper error if the tunnel fails to establish.

---

### 🐛 Bug-04 — Go Deployment Timeout & Inconsistent State

- **Affected Command:** `gy deploy` (Go projects)
- **Description:** The CLI timed out after 15 minutes for a non-web Go deployment. A `resources_pkey` uniqueness violation was also observed in a later attempt. The deployment state was left inconsistent — the app appeared to run (logs were visible) but was never confirmed as deployed.
- **Recommendation:** Improve deployment state reporting, handle database constraint violations gracefully, and surface a clear error when a timeout occurs.

---

### 🐛 Bug-05 — FastAPI Auto-Generated Dockerfile Uses Wrong CMD

- **Affected Command:** `gy deploy` (FastAPI projects)
- **Description:** The auto-generated Dockerfile for FastAPI uses `CMD ["sh", "-c", "python main.py"]`, which does not start a web server. The result is a `502 Bad Gateway` even though the deployment infrastructure succeeds.
- **Recommendation:** Generate a correct Uvicorn command for FastAPI projects:
  ```dockerfile
  CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
  ```

---

### 🔒 Security-01 — Sensitive File Not Auto-Excluded

- **Affected Command:** `gy deploy`
- **Description:** Files such as `.env` and `config.txt` are included in the deployment by default if no `.dockerignore` is present.
- **Impact:** Potential exposure of credentials or other sensitive configuration in the deployed container.
- **Recommendation:** Warn users when sensitive-looking files are present in the build context, or automatically exclude common secret files (`.env`, `*.key`, `*.pem`, etc.) unless explicitly included.

---

### 🔒 Security-02 — Dockerfile `ENV` Secrets Not Detected

- **Affected Command:** `gy deploy`
- **Description:** The CLI does not scan Dockerfiles for hardcoded secrets in `ENV` instructions. Secrets deployed this way are accessible inside the container and may appear in logs.
- **Impact:** Potential exposure of credentials or sensitive values hardcoded in the Dockerfile.
- **Recommendation:** Implement static analysis of the Dockerfile to detect and warn about `ENV` instructions containing secret-like values.

---

### 🔒 Security-03 — AI Secret Detection Not Functional

- **Affected Command:** `gy deploy --ai-key`
- **Description:** The `--ai-key` flag is documented as enabling AI-assisted secret detection, but no warning or detection was observed when deploying a project with a hardcoded API key in source code.
- **Impact:** Users may rely on this feature as a security safeguard, but it provides no protection in its current state.
- **Recommendation:** Fix or fully implement the AI-assisted secret scanning feature, and clearly document its current limitations.

---

## Verdict Summary Table

| Test | Command | Status | Notes |
|---|---|---|---|
| Version | `gy version` | ✅ Pass | Correct version and update status |
| Help | `gy --help` | ✅ Pass | Clear output |
| Signup | `gy signup` | ✅ Pass | Creates account + auto-login |
| Login | `gy login` | ✅ Pass | — |
| Logout | `gy logout` | ⚠️ Non-blocking | Remote sign-out missing `refreshToken` |
| Whoami | `gy whoami` | ✅ Pass | No token leakage |
| List | `gy list` | ✅ Pass | Shows sync status |
| Node.js Deploy | `gy deploy` | ✅ Pass | Requires lockfile + port 8080 |
| Flask Deploy | `gy deploy --port=5000` | ✅ Pass | Port flag required |
| Go Deploy (non-web) | `gy deploy` | ❌ Fail | DB violation / timeout / inconsistent state |
| Go Web Deploy | `gy deploy --port=8000` | ✅ Pass | Auto-detected + deployed |
| FastAPI Deploy | `gy deploy --port=8000` | ⚠️ Partial | Wrong CMD in generated Dockerfile |
| Tunnel | `gy tunnel start` | ❌ Fail | Wrong port + empty URL + false live status |
| Delete Project | `gy delete project` | ✅ Pass | Confirmation prompt works |
| Delete App | `gy delete app` | ⚠️ Partial | Fails on duplicate names |
| Secret in CLI Output | `gy deploy --debug` | ✅ Pass | No leakage in output |
| Sensitive File Inclusion | `gy deploy --debug` | ❌ Fail | `.env` included without `.dockerignore` |
| Git Metadata Exposure | `gy deploy --debug` | ✅ Pass | `.git` excluded |
| Dockerfile Secret Detection | `gy deploy --debug` | ❌ Fail | No warning for `ENV` secrets |
| AI Secret Detection | `gy deploy --debug --ai-key` | ❌ Fail | Feature non-functional |