Ghyamah-task-1-cli2/Readme.md

Ghaymah CLI — Manual Test Report

CLI Version: gy v0.0.24
Test Date: 2026-08-21
Tester: Mahmoud

---

Table of Contents

1. Overview
2. Test Summary
3. Core Commands
   • version
   • help
   • signup
   • login
   • logout
   • whoami
   • list
4. Deployment Tests
   • Node.js
   • Flask
   • Go (Non-Web)
   • Go (Web)
   • FastAPI
5. Tunneling
6. Delete Operations
7. Security Tests
   • Secret Exposure in CLI Output
   • Sensitive File Inclusion
   • Git Metadata Exposure
   • Dockerfile Hardcoded Secret Detection
   • AI-Assisted Secret Detection
8. Known Issues & Bugs
9. Verdict Summary Table

---

Overview

This document reports the results of manual functional and security testing performed on the Ghaymah CLI (gy). Tests cover core authentication commands, multi-language deployment scenarios, tunneling, resource management, and several security-focused scenarios including secret exposure and file inclusion.

---

Test Summary

Category	Total Tests	✅ Pass	⚠️ Partial	❌ Fail
Core Commands	7	6	0	1
Deployments	5	3	1	1
Tunneling	1	0	0	1
Delete Operations	2	1	1	0
Security Tests	5	2	0	3
Total	20	12	2	5

---

Core Commands

1. Version

Field	Details
Command	gy version
Expected	Display current CLI version and update status
Result	✅ Pass

Output:

Ghaymah CLI v0.0.24
You're up to date!

The CLI correctly reports its current version and confirms it is up to date.

---

2. Help

Field	Details
Command	gy --help
Expected	Display CLI description, available commands, usage examples, and global flags
Result	✅ Pass

The help output is clear and provides all necessary information about the available commands and CLI options.

---

3. Signup

Field	Details
Command	gy signup
Expected	Create a new account and auto-login
Result	✅ Pass

Output:

✅ Account created! Logged in as username@gmail.com

Note: An informational log config file not found, using defaults appeared during signup. This is expected behavior and is not an issue.

---

4. Login

Field	Details
Command	gy login
Expected	Authenticate with existing Ghaymah account credentials
Result	✅ Pass

Login succeeds using the email associated with the Ghaymah account. If no account exists, gy signup should be used instead.

---

5. Logout

Field	Details
Command	gy logout
Expected	Sign out and clear local session
Result	⚠️ Non-blocking issue

Output:

Remote sign-out call failed with 400 error: "refreshToken" is required
✅ Logged out

Logout succeeds locally and clears the local config. However, the remote sign-out request fails because the refreshToken is not being included in the request body.

Bug: The API call for remote sign-out is missing the required refreshToken field. The session may remain active server-side until token expiry.

---

6. Whoami

Field	Details
Command	gy whoami
Expected	Display the currently authenticated user's email and User ID
Result	✅ Pass

Output:

✅ Logged in as username@gmail.com
User ID: <id>

No authentication tokens or secrets are exposed. The command correctly identifies the logged-in user.

---

7. List

Field	Details
Command	gy list
Expected	List all projects associated with the authenticated account
Result	✅ Pass

Output:

[proj] synced ghaymah-exam-mahmoud-secops

Projects are listed with their synchronization status as expected.

---

Deployment Tests

Node.js Deployment

Field	Details
Command	gy deploy
Requirements	package.json, package-lock.json, HTTP server listening on port 8080
Result	✅ Pass

Steps:

1. Created a minimal Node.js Hello World project.
2. Initial attempt without a lockfile failed with an npm ci error → generated package-lock.json.
3. First runtime test failed because index.js was missing and later returned 502 because the app wasn't listening on port 8080.
4. Updated index.js to start an HTTP server on port 8080, redeployed, and verified using gy config, gy logs, and the deployed URL.

Output:

Hello from Node.js!

Note: The project must include a package-lock.json and the application must listen on port 8080 (or the configured port) for deployment to succeed.

---

Flask Deployment

Field	Details
Command	gy deploy --port=5000
Requirements	Flask application listening on the configured port
Result	✅ Pass

Steps:

1. Initial deployment using the default port 8080 resulted in 502 Bad Gateway.
2. Changed the application to listen on port 5000.
3. Redeployed with gy deploy --port=5000 — deployment completed successfully.

Note: The --port flag must match the port the application actually listens on.

---

Go (Non-Web) Deployment

Field	Details
Command	gy deploy
Requirements	Valid Go project
Result	❌ Fail

Steps:

1. Deployed a simple Go program that prints Hello from Go! — not a web server.
2. The CLI generated a Dockerfile, built, and uploaded the artifact successfully.
3. However, resource creation failed with a resources_pkey uniqueness violation.
4. In an earlier attempt, the CLI waited for 15 minutes before timing out, while gy logs go-test showed Hello from Go!.

Output:

resources_pkey uniqueness violation
(earlier) Deployment timeout after 15 minutes
Hello from Go!  ← from logs

Bug: The CLI successfully detects and builds the Go application, but deployment does not complete and leaves the state inconsistent. See Known Issues.

---

Go (Web) Deployment

Field	Details
Command	gy deploy --port=8000
Requirements	Go HTTP server listening on the configured port
Result	✅ Pass

Steps:

1. Created a minimal Go HTTP server using net/http listening on port 8000.
2. Deployed without providing a Dockerfile — the CLI auto-detected Go and generated one.
3. Deployment completed and the app was reachable.

Output:

Hello from Go Web!

The CLI correctly detects, builds, and deploys a Go web application when the app listens on the configured port.

---

FastAPI Deployment

Field	Details
Command	gy deploy --port=8000
Requirements	FastAPI application with Uvicorn listening on 0.0.0.0:8000
Result	⚠️ Partial — CLI Issue

Steps:

1. Deployed a FastAPI application using the CLI-generated Dockerfile.
2. Deployment completed successfully, but accessing the URL returned 502 Bad Gateway.
3. Investigation revealed the auto-generated Dockerfile used CMD ["sh", "-c", "python main.py"].
4. Since main.py only defines the FastAPI app object and does not start a server, the container never listens on the port.
5. Verified locally using uvicorn main:app --host 0.0.0.0 --port 8000 — worked correctly.

Output:

Deployment completed successfully.
URL: 502 Bad Gateway

Bug: The auto-generated Dockerfile for FastAPI uses python main.py instead of a Uvicorn command. The correct CMD should be:

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

---

Tunneling

Field	Details
Command	gy tunnel start <name>
Requirements	A successfully deployed application with a configured port
Result	❌ Fail

Steps:

1. Started the tunnel for a deployed application.
2. The CLI reported the tunnel as live but the endpoint was unreachable.
3. The tunnel incorrectly targeted localhost:8080 instead of the configured application port.
4. The tunnel URL returned was empty: https://.

Output:

Tunnel is live at: https://

Bug: Three separate issues were identified (see Known Issues):

• Hardcoded port 8080 instead of using the configured application port.
• Empty tunnel URL returned.
• Tunnel reported as live despite being unreachable.

---

Delete Operations

Delete Project

Field	Details
Command	gy delete project ghaymah-exam-mahmoud-secops
Result	✅ Pass

Output:

✅ Deleted project 'ghaymah-exam-mahmoud-secops'

A confirmation prompt appeared before deletion. The project and all its associated apps were deleted successfully.

---

Delete App

Field	Details
Command	gy delete app <name>
Result	⚠️ Works for unique names; ambiguous for duplicates

App deletion works correctly when the app name is unique. However, if duplicate app names exist, the CLI cannot uniquely identify the target and the deletion becomes unreliable.

Bug: The CLI relies solely on the friendly name for app deletion. Duplicate names should either be prevented or the CLI should support deletion by unique resource ID or interactive selection.

---

Security Tests

Secret Exposure in CLI Output

Field	Details
Command	gy deploy --debug
Test	Checked whether fake credentials in .env and config.txt appear in CLI output or app logs
Result	✅ Pass

Deployment completed successfully. The CLI output and application logs did not contain the test credentials.

---

Sensitive File Inclusion

Field	Details
Command	gy deploy --debug
Test	Verified whether .dockerignore rules are honored and sensitive files are excluded
Result	✅ Pass (with .dockerignore) / ❌ Fail (without .dockerignore)

With .dockerignore: Created .dockerignore rules excluding .env and config.txt. The deployed container contained only the allowed files.

total 16
.dockerignore
Dockerfile

Without .dockerignore: Deployed a project containing .env and config.txt without a .dockerignore. Both files were present in the deployed container.

.env
Dockerfile
config.txt

Security Issue: Sensitive files are not automatically excluded from the deployment. Developers must manually configure .dockerignore. The CLI should warn users when sensitive-looking files (e.g., .env, *.key, config.txt) are included in the build context.

---

Git Metadata Exposure

Field	Details
Command	gy deploy --debug
Test	Verified whether .git directory and metadata are included in the deployed container
Result	✅ Pass

Created a Git repository and committed project files. Added a Dockerfile that checks for .git and .git/config. The deployed container did not contain the .git directory.

Output:

=== GIT CHECK ===
ls: .git: No such file or directory
=== CONFIG CHECK ===
cat: can't open '.git/config': No such file or directory

---

Dockerfile Hardcoded Secret Detection

Field	Details
Command	gy deploy --debug
Test	Verified whether the CLI warns about hardcoded secrets in ENV instructions
Result	❌ Fail

Created a Dockerfile with ENV TEST_SECRET="FAKE_SECRET_12345". Deployed the application and inspected the output and logs.

Output:

Application started
TEST_SECRET=FAKE_SECRET_12345

The CLI deployed the application without any security warning. The hardcoded secret was accessible inside the running container and appeared in the application logs.

Security Issue: The CLI does not scan or warn about hardcoded secrets in ENV instructions within the Dockerfile.

---

AI-Assisted Secret Detection

Field	Details
Command	gy deploy --debug --ai-key="$GY_AI_KEY"
Test	Verified whether the --ai-key flag enables detection of hardcoded secrets in source code
Result	❌ Fail

Created a test project with a hardcoded API key in app.sh. Deployed with --ai-key flag enabled. No security warning or detection message was observed.

Output:

Application started

Security Issue: The AI-assisted secret detection feature (--ai-key) did not identify or warn about the hardcoded API key. This feature appears to be non-functional or not yet implemented.

---

Known Issues & Bugs

🐛 Bug-01 — Duplicate App Names Cause Ambiguity

• Affected Command: gy delete app <name>
• Description: The CLI identifies apps by friendly name only. When multiple apps share the same name, the CLI cannot uniquely target an app for deletion.
• Recommendation: Prevent duplicate app names at creation time, or support deletion by unique resource ID / interactive disambiguation.

---

🐛 Bug-02 — Default Port Assumption Causes 502 Errors

• Affected Commands: gy deploy, gy tunnel start
• Description: The CLI assumes port 8080 by default. Applications listening on a different port will result in 502 Bad Gateway.
• Recommendation: Detect the actual application port automatically, or require explicit port configuration with a clearer error message.

---

🐛 Bug-03 — Tunnel Fails Completely

• Affected Command: gy tunnel start <name>
• Description: Three compounding issues:
  1. Port is hardcoded to 8080 instead of using the configured application port.
  2. The tunnel URL is returned empty (https://).
  3. The tunnel is reported as live despite being unreachable.
• Recommendation: Fix port synchronization with deployment config, validate the tunnel URL before reporting success, and surface a proper error if the tunnel fails to establish.

---

🐛 Bug-04 — Go Deployment Timeout & Inconsistent State

• Affected Command: gy deploy (Go projects)
• Description: The CLI timed out after 15 minutes for a non-web Go deployment. A resources_pkey uniqueness violation was also observed in a later attempt. The deployment state was left inconsistent — the app appeared to run (logs were visible) but was never confirmed as deployed.
• Recommendation: Improve deployment state reporting, handle database constraint violations gracefully, and surface a clear error when a timeout occurs.

---

🐛 Bug-05 — FastAPI Auto-Generated Dockerfile Uses Wrong CMD

• Affected Command: gy deploy (FastAPI projects)
• Description: The auto-generated Dockerfile for FastAPI uses CMD ["sh", "-c", "python main.py"], which does not start a web server. The result is a 502 Bad Gateway even though the deployment infrastructure succeeds.
• Recommendation: Generate a correct Uvicorn command for FastAPI projects:

  CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
  

---

🔒 Security-01 — Sensitive File Not Auto-Excluded

• Affected Command: gy deploy
• Description: Files such as .env and config.txt are included in the deployment by default if no .dockerignore is present.
• Impact: Potential exposure of credentials or other sensitive configuration in the deployed container.
• Recommendation: Warn users when sensitive-looking files are present in the build context, or automatically exclude common secret files (.env, *.key, *.pem, etc.) unless explicitly included.

---

🔒 Security-02 — Dockerfile ENV Secrets Not Detected

• Affected Command: gy deploy
• Description: The CLI does not scan Dockerfiles for hardcoded secrets in ENV instructions. Secrets deployed this way are accessible inside the container and may appear in logs.
• Impact: Potential exposure of credentials or sensitive values hardcoded in the Dockerfile.
• Recommendation: Implement static analysis of the Dockerfile to detect and warn about ENV instructions containing secret-like values.

---

🔒 Security-03 — AI Secret Detection Not Functional

• Affected Command: gy deploy --ai-key
• Description: The --ai-key flag is documented as enabling AI-assisted secret detection, but no warning or detection was observed when deploying a project with a hardcoded API key in source code.
• Impact: Users may rely on this feature as a security safeguard, but it provides no protection in its current state.
• Recommendation: Fix or fully implement the AI-assisted secret scanning feature, and clearly document its current limitations.

---

Verdict Summary Table

Test	Command	Status	Notes
Version	gy version	✅ Pass	Correct version and update status
Help	gy --help	✅ Pass	Clear output
Signup	gy signup	✅ Pass	Creates account + auto-login
Login	gy login	✅ Pass	—
Logout	gy logout	⚠️ Non-blocking	Remote sign-out missing refreshToken
Whoami	gy whoami	✅ Pass	No token leakage
List	gy list	✅ Pass	Shows sync status
Node.js Deploy	gy deploy	✅ Pass	Requires lockfile + port 8080
Flask Deploy	gy deploy --port=5000	✅ Pass	Port flag required
Go Deploy (non-web)	gy deploy	❌ Fail	DB violation / timeout / inconsistent state
Go Web Deploy	gy deploy --port=8000	✅ Pass	Auto-detected + deployed
FastAPI Deploy	gy deploy --port=8000	⚠️ Partial	Wrong CMD in generated Dockerfile
Tunnel	gy tunnel start	❌ Fail	Wrong port + empty URL + false live status
Delete Project	gy delete project	✅ Pass	Confirmation prompt works
Delete App	gy delete app	⚠️ Partial	Fails on duplicate names
Secret in CLI Output	gy deploy --debug	✅ Pass	No leakage in output
Sensitive File Inclusion	gy deploy --debug	❌ Fail	.env included without .dockerignore
Git Metadata Exposure	gy deploy --debug	✅ Pass	.git excluded
Dockerfile Secret Detection	gy deploy --debug	❌ Fail	No warning for ENV secrets
AI Secret Detection	gy deploy --debug --ai-key	❌ Fail	Feature non-functional