diff --git a/README.md b/README.md index 5817980..deafdfa 100644 --- a/README.md +++ b/README.md @@ -1,369 +1,369 @@ -# 🚀 LLM-Powered URL Health Checker Generator - -[![Python](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://python.org) -[![LLM APIs](https://img.shields.io/badge/LLM-Multi--Provider-green.svg)](https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator/src/branch/main) -[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -[![Jupyter](https://img.shields.io/badge/Jupyter-Notebook-orange.svg)](https://jupyter.org) - -🤖 **AI-Powered Code Generation** | 🌐 **URL Health Monitoring** | 📊 **DevOps Tooling** | ⚡ **Automated CLI Tools** - -> **Transform ideas into code with ANY LLM API!** This project showcases how to harness various Large Language Model APIs (OpenAI, Anthropic Claude, Google Gemini, Azure, local models, and more) to automatically generate production-ready Python CLI tools. Watch as AI creates a sophisticated URL health monitoring system that rivals hand-coded solutions. - -## 🎯 What This Project Does - -This project demonstrates cutting-edge AI-assisted development by: -- 🧠 **Intelligently generating** complete Python applications using any LLM API -- 🔍 **Creating a professional URL health checker** with advanced monitoring capabilities -- 📈 **Showcasing practical AI applications** in DevOps and system administration -- 🛠️ **Providing a universal framework** for AI-powered code generation across multiple providers - -## 🌟 Supported LLM Providers - -| Provider | Models | Configuration | -|----------|---------|--------------| -| **OpenAI** | GPT-4, GPT-3.5-Turbo | Standard OpenAI API | -| **Anthropic** | Claude 3 Opus, Sonnet, Haiku | Claude API | -| **Google** | Gemini Pro, Gemini Ultra | Google AI Studio | -| **Azure OpenAI** | GPT-4, GPT-3.5 | Azure OpenAI Service | -| **Cohere** | Command, Generate | Cohere API | -| **Hugging Face** | Various open models | Inference API | -| **Local Models** | LLaMA, Mistral, etc. | Ollama, LM Studio | -| **Custom APIs** | Any OpenAI-compatible | Custom endpoints | - -## 🚀 Features - -### Core Framework -- **Universal LLM Support**: Works with any OpenAI-compatible API endpoint -- **Provider Flexibility**: Easy switching between different LLM providers -- **Smart Configuration Management**: Secure environment variable handling with validation -- **AI-Powered Code Generation**: Uses your choice of LLM to generate complete, functional Python scripts -- **Modular Design**: Clean, reusable configuration class for API integration -- **Error Handling**: Robust validation and error management across providers - -### Generated URL Health Checker -- **Batch URL Testing**: Process multiple URLs from input files -- **Performance Monitoring**: Measure and report response times -- **Status Code Tracking**: Monitor HTTP response codes -- **Configurable Parameters**: Customizable timeouts and retry mechanisms -- **CSV Export**: Structured data output for analysis -- **Summary Statistics**: Real-time reporting of test results -- **CLI Interface**: Professional command-line tool with argument parsing - -## ⚡ Quick Start - -```bash -# 1. Clone and setup -git clone https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator.git -cd LLM-Powered-URL-Health-Checker-Generator - -# 2. Install dependencies -pip install openai anthropic google-generativeai python-dotenv requests - -# 3. Configure environment (see provider examples below) -cp .env.example .env -# Edit .env with your preferred provider - -# 4. Run the notebook to generate the tool -jupyter notebook Final.ipynb - -# 5. Use the generated URL checker -echo "https://github.com" > urls.txt -python url_checker.py --input urls.txt --output results.csv -``` - -## 📋 Prerequisites - -- Python 3.10+ -- Access to at least one LLM API (see supported providers) -- Required Python packages (see installation) - -## 🔧 Detailed Installation - -### 1. **Clone the Repository** - ```bash - git clone https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator.git - cd LLM-Powered-URL-Health-Checker-Generator - ``` - -### 2. **Set Up Python Environment** (Recommended) - ```bash - # Create virtual environment - python -m venv venv - - # Activate it (Windows) - venv\Scripts\activate - # Or on Linux/Mac - source venv/bin/activate - ``` - -### 3. **Install Dependencies** - ```bash - # Core dependencies - pip install python-dotenv requests jupyter - - # Install based on your provider: - pip install openai # For OpenAI - pip install anthropic # For Claude - pip install google-generativeai # For Gemini - # Or install all: - pip install -r requirements.txt - ``` - -### 4. **Configure Environment Variables** - -Create a `.env` file based on your chosen provider: - -#### OpenAI Configuration -```env -LLM_PROVIDER=openai -OPENAI_API_KEY=sk-your_api_key_here -BASE_URL=https://api.openai.com/v1 -MODEL_NAME=gpt-4 -``` - -#### Anthropic Claude Configuration -```env -LLM_PROVIDER=anthropic -ANTHROPIC_API_KEY=sk-ant-your_api_key_here -BASE_URL=https://api.anthropic.com -MODEL_NAME=claude-3-opus-20240229 -``` - -#### Google Gemini Configuration -```env -LLM_PROVIDER=google -GOOGLE_API_KEY=your_api_key_here -MODEL_NAME=gemini-pro -``` - -#### Azure OpenAI Configuration -```env -LLM_PROVIDER=azure -AZURE_API_KEY=your_api_key_here -BASE_URL=https://your-resource.openai.azure.com -AZURE_DEPLOYMENT_NAME=your-deployment -MODEL_NAME=gpt-4 -``` - -#### Local Models (Ollama) Configuration -```env -LLM_PROVIDER=ollama -BASE_URL=http://localhost:11434/v1 -MODEL_NAME=llama2:13b -# No API key needed for local models -``` - -#### Custom OpenAI-Compatible API -```env -LLM_PROVIDER=custom -API_KEY=your_api_key_here -BASE_URL=https://your-custom-endpoint.com/v1 -MODEL_NAME=your-model-name -``` - -## 🎯 Usage Guide - -### Step 1: Generate the URL Checker -1. **Open the Jupyter Notebook:** - ```bash - jupyter notebook Final.ipynb - ``` - -2. **Select your LLM provider** in the configuration cell - -3. **Execute all cells** to: - - Configure your chosen LLM API connection - - Generate the URL health checker script - - Save it as `url_checker.py` - -### Step 2: Prepare Your URL List -Create a `urls.txt` file with websites to monitor: -```text -https://www.google.com -https://github.com -https://stackoverflow.com -https://httpbin.org/status/200 -https://httpbin.org/delay/3 -# Comments are ignored -https://nonexistent-website-12345.com -``` - -### Step 3: Run the Health Checker -```bash -# Basic usage -python url_checker.py - -# Custom parameters -python url_checker.py --input my_sites.txt --output health_report.csv --timeout 10 --retries 5 -``` - -## 📊 Example Output - -### Console Output -``` -🔍 URL Health Checker Starting... -✅ https://www.google.com - 200 OK (156ms) -✅ https://github.com - 200 OK (245ms) -❌ https://nonexistent-site.com - Connection failed -📊 Summary: 2/3 successful (66.7%), Average: 200ms -``` - -### CSV Report (`results.csv`) -```csv -url,status_code,response_time_ms,success -https://www.google.com,200,156.3,True -https://github.com,200,245.7,True -https://nonexistent-site.com,0,0.0,False -``` - -## 🛠️ Command Line Options - -| Option | Default | Description | Example | -|--------|---------|-------------|---------| -| `--input` | `urls.txt` | Input file containing URLs | `--input websites.txt` | -| `--output` | `results.csv` | Output CSV file for results | `--output report.csv` | -| `--timeout` | `5` | Request timeout in seconds | `--timeout 10` | -| `--retries` | `3` | Number of retry attempts | `--retries 5` | - -## 🏗️ Project Architecture - -``` -LLM-Powered-URL-Health-Checker-Generator/ -├── 📓 Final.ipynb # Main LLM integration notebook -├── 📄 README.md # This documentation -├── 🐍 url_checker.py # Generated URL health checker -├── ⚙️ .env # Environment variables (create this) -├── ⚙️ .env.example # Example environment configuration -├── 📝 urls.txt # Input URLs list -├── 📊 results.csv # Output results -├── 📦 requirements.txt # Python dependencies -└── 📜 .gitignore # Git ignore file -``` - -## 🔐 Security & Best Practices - -### Environment Security -- ✅ **Never commit `.env` files** - Add to `.gitignore` -- ✅ **Use environment variables** for all API keys -- ✅ **Rotate API keys regularly** -- ✅ **Monitor API usage** and set billing alerts -- ✅ **Use provider-specific security features** (API key restrictions, rate limits) - -### Code Safety -- ✅ **Review generated code** before execution -- ✅ **Test with safe URLs** first -- ✅ **Use virtual environments** to isolate dependencies -- ✅ **Validate input files** before processing - - - -## 🤖 LLM Integration Details - -### Model Configuration Examples - -#### OpenAI GPT-4 -```python -model="gpt-4" -temperature=0.2 # Low for consistent code -max_tokens=8000 # Balanced for complete scripts -``` - -#### Claude 3 Opus -```python -model="claude-3-opus-20240229" -temperature=0.2 -max_tokens=4000 -``` - -#### Gemini Pro -```python -model="gemini-pro" -temperature=0.2 -max_tokens=8192 -``` - -#### Local Models (Ollama) -```python -model="codellama:13b" -temperature=0.1 # Very low for code consistency -max_tokens=4096 -``` - -### Prompt Engineering -The project uses carefully crafted prompts optimized for each provider to ensure: -- 🎯 **Focused output** - Only relevant code generation -- 🔧 **Practical functionality** - Real-world utility -- 📚 **Documentation** - Self-explaining code -- ⚡ **Efficiency** - Optimized performance -- 🔄 **Provider compatibility** - Works across different LLMs - -## 📈 Use Cases - -### DevOps & Monitoring -- **Website uptime monitoring** -- **API endpoint health checks** -- **Load balancer status verification** -- **CDN performance testing** -- **Multi-region availability checking** - -### Development & Testing -- **Integration test automation** -- **Environment validation** -- **Dependency verification** -- **Service discovery** -- **CI/CD pipeline validation** - - -## 🤝 Contributing - -We welcome contributions! Here's how to get started: - -### 1. Fork & Clone -```bash -# Fork the repository on GitPasha first, then: -git clone https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator.git -``` - -### 2. Create Feature Branch -```bash -git checkout -b feature/amazing-new-feature -``` -### 3. Make Changes -- Add support for new LLM providers -- Improve prompt engineering -- Add new tool generation capabilities -- Update documentation -- Add tests if applicable - -### 4. Submit Pull Request -```bash -git add . -git commit -m "✨ Add amazing new feature" -git push origin feature/amazing-new-feature -``` - - -## 🙏 Acknowledgments - -- **LLM Providers** - OpenAI, Anthropic, Google, and the open-source community -- **Python Community** - For excellent libraries (`requests`, `python-dotenv`) -- **Jupyter Project** - For the interactive development environment -- **Open Source Models** - LLaMA, Mistral, and other contributors -- **Contributors** - Everyone who helps improve this project - -## 📞 Support & Community - -### Getting Help -- 🐛 **Bug Reports**: [GitPasha Issues](https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator/issues) -- 💡 **Feature Requests**: [GitPasha Discussions](https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator/discussions) -- 📖 **Documentation**: This README and notebook comments -- 💬 **Community**: Join our discussions for tips and tricks - -### Troubleshooting -| Problem | Solution | -|---------|----------| -| `ModuleNotFoundError` | Run `pip install -r requirements.txt` | -| `API Key Error` | Check your `.env` file and provider configuration | -| `Timeout Issues` | Increase `--timeout` parameter or switch providers | -| `Generation Fails` | Verify API credits, permissions, and model availability | +# 🚀 LLM-Powered URL Health Checker Generator + +[![Python](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://python.org) +[![LLM APIs](https://img.shields.io/badge/LLM-Multi--Provider-green.svg)](https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator/src/branch/main) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) +[![Jupyter](https://img.shields.io/badge/Jupyter-Notebook-orange.svg)](https://jupyter.org) + +🤖 **AI-Powered Code Generation** | 🌐 **URL Health Monitoring** | 📊 **DevOps Tooling** | ⚡ **Automated CLI Tools** + +> **Transform ideas into code with ANY LLM API!** This project showcases how to harness various Large Language Model APIs (OpenAI, Anthropic Claude, Google Gemini, Azure, local models, and more) to automatically generate production-ready Python CLI tools. Watch as AI creates a sophisticated URL health monitoring system that rivals hand-coded solutions. + +## 🎯 What This Project Does + +This project demonstrates cutting-edge AI-assisted development by: +- 🧠 **Intelligently generating** complete Python applications using any LLM API +- 🔍 **Creating a professional URL health checker** with advanced monitoring capabilities +- 📈 **Showcasing practical AI applications** in DevOps and system administration +- 🛠️ **Providing a universal framework** for AI-powered code generation across multiple providers + +## 🌟 Supported LLM Providers + +| Provider | Models | Configuration | +|----------|---------|--------------| +| **OpenAI** | GPT-4, GPT-3.5-Turbo | Standard OpenAI API | +| **Anthropic** | Claude 3 Opus, Sonnet, Haiku | Claude API | +| **Google** | Gemini Pro, Gemini Ultra | Google AI Studio | +| **Azure OpenAI** | GPT-4, GPT-3.5 | Azure OpenAI Service | +| **Cohere** | Command, Generate | Cohere API | +| **Hugging Face** | Various open models | Inference API | +| **Local Models** | LLaMA, Mistral, etc. | Ollama, LM Studio | +| **Custom APIs** | Any OpenAI-compatible | Custom endpoints | + +## 🚀 Features + +### Core Framework +- **Universal LLM Support**: Works with any OpenAI-compatible API endpoint +- **Provider Flexibility**: Easy switching between different LLM providers +- **Smart Configuration Management**: Secure environment variable handling with validation +- **AI-Powered Code Generation**: Uses your choice of LLM to generate complete, functional Python scripts +- **Modular Design**: Clean, reusable configuration class for API integration +- **Error Handling**: Robust validation and error management across providers + +### Generated URL Health Checker +- **Batch URL Testing**: Process multiple URLs from input files +- **Performance Monitoring**: Measure and report response times +- **Status Code Tracking**: Monitor HTTP response codes +- **Configurable Parameters**: Customizable timeouts and retry mechanisms +- **CSV Export**: Structured data output for analysis +- **Summary Statistics**: Real-time reporting of test results +- **CLI Interface**: Professional command-line tool with argument parsing + +## ⚡ Quick Start + +```bash +# 1. Clone and setup +git clone https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator.git +cd LLM-Powered-URL-Health-Checker-Generator + +# 2. Install dependencies +pip install openai anthropic google-generativeai python-dotenv requests + +# 3. Configure environment (see provider examples below) +cp .env.example .env +# Edit .env with your preferred provider + +# 4. Run the notebook to generate the tool +jupyter notebook Final.ipynb + +# 5. Use the generated URL checker +echo "https://github.com" > urls.txt +python url_checker.py --input urls.txt --output results.csv +``` + +## 📋 Prerequisites + +- Python 3.10+ +- Access to at least one LLM API (see supported providers) +- Required Python packages (see installation) + +## 🔧 Detailed Installation + +### 1. **Clone the Repository** + ```bash + git clone https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator.git + cd LLM-Powered-URL-Health-Checker-Generator + ``` + +### 2. **Set Up Python Environment** (Recommended) + ```bash + # Create virtual environment + python -m venv venv + + # Activate it (Windows) + venv\Scripts\activate + # Or on Linux/Mac + source venv/bin/activate + ``` + +### 3. **Install Dependencies** + ```bash + # Core dependencies + pip install python-dotenv requests jupyter + + # Install based on your provider: + pip install openai # For OpenAI + pip install anthropic # For Claude + pip install google-generativeai # For Gemini + # Or install all: + pip install -r requirements.txt + ``` + +### 4. **Configure Environment Variables** + +Create a `.env` file based on your chosen provider: + +#### OpenAI Configuration +```env +LLM_PROVIDER=openai +OPENAI_API_KEY=sk-your_api_key_here +BASE_URL=https://api.openai.com/v1 +MODEL_NAME=gpt-4 +``` + +#### Anthropic Claude Configuration +```env +LLM_PROVIDER=anthropic +ANTHROPIC_API_KEY=sk-ant-your_api_key_here +BASE_URL=https://api.anthropic.com +MODEL_NAME=claude-3-opus-20240229 +``` + +#### Google Gemini Configuration +```env +LLM_PROVIDER=google +GOOGLE_API_KEY=your_api_key_here +MODEL_NAME=gemini-pro +``` + +#### Azure OpenAI Configuration +```env +LLM_PROVIDER=azure +AZURE_API_KEY=your_api_key_here +BASE_URL=https://your-resource.openai.azure.com +AZURE_DEPLOYMENT_NAME=your-deployment +MODEL_NAME=gpt-4 +``` + +#### Local Models (Ollama) Configuration +```env +LLM_PROVIDER=ollama +BASE_URL=http://localhost:11434/v1 +MODEL_NAME=llama2:13b +# No API key needed for local models +``` + +#### Custom OpenAI-Compatible API +```env +LLM_PROVIDER=custom +API_KEY=your_api_key_here +BASE_URL=https://your-custom-endpoint.com/v1 +MODEL_NAME=your-model-name +``` + +## 🎯 Usage Guide + +### Step 1: Generate the URL Checker +1. **Open the Jupyter Notebook:** + ```bash + jupyter notebook Final.ipynb + ``` + +2. **Select your LLM provider** in the configuration cell + +3. **Execute all cells** to: + - Configure your chosen LLM API connection + - Generate the URL health checker script + - Save it as `url_checker.py` + +### Step 2: Prepare Your URL List +Create a `urls.txt` file with websites to monitor: +```text +https://www.google.com +https://github.com +https://stackoverflow.com +https://httpbin.org/status/200 +https://httpbin.org/delay/3 +# Comments are ignored +https://nonexistent-website-12345.com +``` + +### Step 3: Run the Health Checker +```bash +# Basic usage +python url_checker.py + +# Custom parameters +python url_checker.py --input my_sites.txt --output health_report.csv +``` + +## 📊 Example Output + +### Console Output +``` +🔍 URL Health Checker Starting... +✅ https://www.google.com - 200 OK (156ms) +✅ https://github.com - 200 OK (245ms) +❌ https://nonexistent-site.com - Connection failed +📊 Summary: 2/3 successful (66.7%), Average: 200ms +``` + +### CSV Report (`results.csv`) +```csv +url,status_code,response_time_ms,success +https://www.google.com,200,156.3,True +https://github.com,200,245.7,True +https://nonexistent-site.com,0,0.0,False +``` + +## 🛠️ Command Line Options + +| Option | Default | Description | Example | +|--------|---------|-------------|---------| +| `--input` | `urls.txt` | Input file containing URLs | `--input websites.txt` | +| `--output` | `results.csv` | Output CSV file for results | `--output report.csv` | +| `--timeout` | `5` | Request timeout in seconds | `--timeout 10` | +| `--retries` | `3` | Number of retry attempts | `--retries 5` | + +## 🏗️ Project Architecture + +``` +LLM-Powered-URL-Health-Checker-Generator/ +├── 📓 Final.ipynb # Main LLM integration notebook +├── 📄 README.md # This documentation +├── 🐍 url_checker.py # Generated URL health checker +├── ⚙️ .env # Environment variables (create this) +├── ⚙️ .env.example # Example environment configuration +├── 📝 urls.txt # Input URLs list +├── 📊 results.csv # Output results +├── 📦 requirements.txt # Python dependencies +└── 📜 .gitignore # Git ignore file +``` + +## 🔐 Security & Best Practices + +### Environment Security +- ✅ **Never commit `.env` files** - Add to `.gitignore` +- ✅ **Use environment variables** for all API keys +- ✅ **Rotate API keys regularly** +- ✅ **Monitor API usage** and set billing alerts +- ✅ **Use provider-specific security features** (API key restrictions, rate limits) + +### Code Safety +- ✅ **Review generated code** before execution +- ✅ **Test with safe URLs** first +- ✅ **Use virtual environments** to isolate dependencies +- ✅ **Validate input files** before processing + + + +## 🤖 LLM Integration Details + +### Model Configuration Examples + +#### OpenAI GPT-4 +```python +model="gpt-4" +temperature=0.2 # Low for consistent code +max_tokens=8000 # Balanced for complete scripts +``` + +#### Claude 3 Opus +```python +model="claude-3-opus-20240229" +temperature=0.2 +max_tokens=4000 +``` + +#### Gemini Pro +```python +model="gemini-pro" +temperature=0.2 +max_tokens=8192 +``` + +#### Local Models (Ollama) +```python +model="codellama:13b" +temperature=0.1 # Very low for code consistency +max_tokens=4096 +``` + +### Prompt Engineering +The project uses carefully crafted prompts optimized for each provider to ensure: +- 🎯 **Focused output** - Only relevant code generation +- 🔧 **Practical functionality** - Real-world utility +- 📚 **Documentation** - Self-explaining code +- ⚡ **Efficiency** - Optimized performance +- 🔄 **Provider compatibility** - Works across different LLMs + +## 📈 Use Cases + +### DevOps & Monitoring +- **Website uptime monitoring** +- **API endpoint health checks** +- **Load balancer status verification** +- **CDN performance testing** +- **Multi-region availability checking** + +### Development & Testing +- **Integration test automation** +- **Environment validation** +- **Dependency verification** +- **Service discovery** +- **CI/CD pipeline validation** + + +## 🤝 Contributing + +We welcome contributions! Here's how to get started: + +### 1. Fork & Clone +```bash +# Fork the repository on GitPasha first, then: +git clone https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator.git +``` + +### 2. Create Feature Branch +```bash +git checkout -b feature/amazing-new-feature +``` +### 3. Make Changes +- Add support for new LLM providers +- Improve prompt engineering +- Add new tool generation capabilities +- Update documentation +- Add tests if applicable + +### 4. Submit Pull Request +```bash +git add . +git commit -m "✨ Add amazing new feature" +git push origin feature/amazing-new-feature +``` + + +## 🙏 Acknowledgments + +- **LLM Providers** - OpenAI, Anthropic, Google, and the open-source community +- **Python Community** - For excellent libraries (`requests`, `python-dotenv`) +- **Jupyter Project** - For the interactive development environment +- **Open Source Models** - LLaMA, Mistral, and other contributors +- **Contributors** - Everyone who helps improve this project + +## 📞 Support & Community + +### Getting Help +- 🐛 **Bug Reports**: [GitPasha Issues](https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator/issues) +- 💡 **Feature Requests**: [GitPasha Discussions](https://app.gitpasha.com/MahmoudZaafan/LLM-Powered-URL-Health-Checker-Generator/discussions) +- 📖 **Documentation**: This README and notebook comments +- 💬 **Community**: Join our discussions for tips and tricks + +### Troubleshooting +| Problem | Solution | +|---------|----------| +| `ModuleNotFoundError` | Run `pip install -r requirements.txt` | +| `API Key Error` | Check your `.env` file and provider configuration | +| `Timeout Issues` | Increase `--timeout` parameter or switch providers | +| `Generation Fails` | Verify API credits, permissions, and model availability | | `Provider Not Working` | Check API end \ No newline at end of file