A comprehensive ML & LLM observability platform for production systems.
Watchtower AI is a self-hosted monitoring framework designed to track model performance, data drift, data quality, and LLM behavior in real-time. It provides actionable insights through an integrated dashboard, enabling data-driven decisions for model retraining and maintenance in production environments.
Think of it as Datadog, but specifically built for ML pipelines and LLM applications.
👉 https://watchtower-ai-production-604f.up.railway.app
| Feature | Description |
|---|---|
| Data Drift Detection | Detect distribution shifts using KS Test, PSI, Mean/Median/Variance Shift, and Model-Based drift detection. |
| Data Quality Monitoring | Validate schema, detect missing values, and identify duplicate records automatically. |
| Prediction Monitoring | Track classification (accuracy, precision, recall, F1, AUC) and regression (MAE, MSE, RMSE, R²) metrics over time. |
| LLM Observability | Monitor prompt-response pairs, detect toxicity, track token usage, and evaluate response quality with an LLM Judge. |
| AI-Powered Insights | Get natural language interpretations of drift results powered by LLM analysis. |
| Real-Time Alerting | Configurable thresholds trigger alerts when drift or quality issues exceed limits. |
| Interactive Dashboard | Visualize trends, drill into per-feature drift snapshots, and compare baseline vs. production distributions. |
┌─────────────────────────────────────────────────────────────┐
│ Watchtower AI Platform │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Frontend │ │ FastAPI │ │ PostgreSQL │ │
│ │ (Jinja2 + │──│ Backend │──│ (Supabase / │ │
│ │ Vanilla JS) │ │ Engine │ │ Render DB) │ │
│ └──────────────┘ └──────┬───────┘ └──────────────┘ │
│ │ │
│ ┌───────────────┼───────────────┐ │
│ │ │ │ │
│ ┌───────▼──────┐ ┌──────▼──────┐ ┌─────▼───────┐ │
│ │ Data Drift │ │ Prediction │ │ LLM │ │
│ │ Detection │ │ Monitoring │ │ Monitoring │ │
│ │ Service │ │ Service │ │ Service │ │
│ └──────────────┘ └─────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ Python SDK (pip install) │
├──────────────┬──────────────────┬────────────────────────────┤
│ InputMonitor │ ModelMonitor │ LLMMonitor │
│ (Tabular) │ (Predictions) │ (Prompt/Response) │
└──────────────┴──────────────────┴────────────────────────────┘
pip install watchtower-sdkimport pandas as pd
from watchtower.monitor import WatchtowerInputMonitor
monitor = WatchtowerInputMonitor(
project_name="My ML Project",
api_key="your_project_api_key",
endpoint="https://watchtower-ai-production-604f.up.railway.app"
)
df = pd.read_csv("production_data.csv")
response = monitor.log(df)
print(response)from watchtower.monitor import WatchtowerModelMonitor
model_monitor = WatchtowerModelMonitor(
project_name="Fraud Detector",
api_key="your_project_api_key",
endpoint="https://watchtower-ai-production-604f.up.railway.app",
model_type="classification"
)
model_monitor.log(
predictions=[0, 1, 0, 0, 1, 1, 0, 1],
accuracy=0.92,
precision=0.89,
recall=0.95,
f1_score=0.91,
roc_auc=0.96
)from watchtower.llm_monitor import WatchtowerLLMMonitor
llm_monitor = WatchtowerLLMMonitor(
api_key="your_api_key",
project_name="Customer Support Bot",
endpoint="https://watchtower-ai-production-604f.up.railway.app"
)
llm_monitor.log_interaction(
input_text="How do I reset my password?",
response_text="Go to Settings > Security > Reset Password.",
metadata={"model": "gpt-4", "latency_ms": 320}
)Watchtower runs 6 statistical tests on every monitoring snapshot:
| Test | Type | What It Detects |
|---|---|---|
| Mean Shift | Statistical | Change in central tendency ( >10% relative change) |
| Median Shift | Statistical | Robust central tendency change (outlier resistant) |
| Variance Shift | Statistical | Change in data spread/dispersion ( >20% relative change) |
| KS Test | Distribution | Full distribution comparison (p-value < 0.05) |
| PSI | Distribution | Population stability (Low < 0.1, Moderate 0.1–0.25, High > 0.25) |
| Model-Based | ML | RandomForest classifier distinguishes baseline from current data |
All thresholds are configurable per-project via the dashboard.
- Python 3.8+
- PostgreSQL database (Supabase, Render DB, or any PostgreSQL instance)
- A Groq API key (for LLM-powered insights)
# Clone the repository
git clone https://github.com/aniq63/Watchtower-AI.git
cd Watchtower-AI
# Create virtual environment
python -m venv .myenv
.myenv\Scripts\activate # Windows
# source .myenv/bin/activate # macOS/Linux
# Install dependencies
pip install -r requirements.txt
# Set environment variables
set GROQ_API_KEY=your_groq_key
set DATABASE_URL=postgresql+asyncpg://user:pass@host:port/db
# Run the server
uvicorn main:app --reloaddocker-compose up --build- Push your code to GitHub.
- Create a new Web Service on Railway.
- Connect your GitHub repository.
- Set environment variables (
DATABASE_URL,GROQ_API_KEY). - Railway will auto-build from the
Dockerfileand deploy.
Watchtower-AI/
├── app/
│ ├── config.py # Environment & settings
│ ├── constants.py # Default thresholds & configs
│ ├── database/ # SQLAlchemy models & connection
│ ├── routes/ # FastAPI route handlers
│ │ ├── ingest.py # Data ingestion endpoint
│ │ ├── drift_detection.py # Drift analysis endpoints
│ │ ├── data_quality.py # Quality check endpoints
│ │ ├── prediction_monitoring.py
│ │ └── llm_monitoring.py
│ └── services/ # Core business logic
│ ├── feature_monitoring/ # Drift detection engine
│ ├── prediction_monitoring/# Prediction drift analysis
│ └── llm_monitoring/ # LLM evaluation engine
├── frontend/
│ ├── templates/ # Jinja2 HTML templates
│ └── static/ # CSS, JS, assets
├── watchtower_sdk/ # Python SDK (published to PyPI)
│ └── watchtower/
│ ├── client.py # HTTP client
│ ├── monitor.py # InputMonitor + ModelMonitor
│ └── llm_monitor.py # LLMMonitor
├── Dockerfile # Production container
├── docker-compose.yml # Local dev orchestration
├── requirements.txt # Python dependencies
└── main.py # FastAPI entrypoint
| SDK Class | Purpose | Endpoint |
|---|---|---|
WatchtowerInputMonitor |
Log tabular/feature data | /ingest |
WatchtowerModelMonitor |
Log predictions & metrics | /ingest/predictions |
WatchtowerLLMMonitor |
Log LLM prompt-response pairs | /llm/ingest |
Full documentation: https://watchtower-ai-production-604f.up.railway.app/documentation
PyPI Package: https://pypi.org/project/watchtower-sdk/
| Layer | Technology |
|---|---|
| Backend | FastAPI, Uvicorn, SQLAlchemy (async) |
| Database | PostgreSQL (asyncpg), Supabase compatible |
| Frontend | Jinja2, Vanilla JS, CSS |
| ML/Stats | Pandas, NumPy, SciPy, Scikit-learn |
| LLM | LangChain, Groq, Detoxify |
| Deployment | Docker, Railway.app |
| SDK | Published on PyPI |
Watchtower AI is open-source and we welcome contributions from the community! Here's how you can help:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
- Bug Reports — Found a bug? Open an issue.
- Feature Requests — Have an idea? Start a discussion.
- Documentation — Improve docs, fix typos, add examples.
- Tests — Add unit tests or integration tests.
- Code — Fix bugs, optimize performance, or add new drift detection algorithms.
All contributions, big or small, are greatly appreciated!
This project is licensed under the MIT License — see the LICENSE file for details.
Muhammad Aniq Ramzan
- 📧 Email: aniqramzan5758@gmail.com
- GitHub: @aniq63
Built with ❤️ for the ML community