diff --git a/.env.example b/.env.example index b07806d5..a0db8dd6 100644 --- a/.env.example +++ b/.env.example @@ -1,6 +1,9 @@ # Required JWT_SECRET_KEY="your-super-secret-key-change-in-production" SESSION_SECRET_KEY="your-session-secret-key-change-in-production" + +# Direct/local runs can set POSTGRES_URI here. Docker Compose sets the container +# POSTGRES_URI in docker-compose.yml/docker-compose.run.yml. POSTGRES_URI="postgresql+asyncpg://morphik:morphik@localhost:5432/morphik" # LLM Provider API Keys (set the ones you use) @@ -18,3 +21,6 @@ TURBOPUFFER_API_KEY= SENTRY_DSN= LOCAL_URI_PASSWORD= LITELLM_DUMMY_API_KEY= + +# Uncomment and set to false before first start to disable self-hosted telemetry. +# TELEMETRY=false diff --git a/DOCKER.md b/DOCKER.md index a9e02c75..57575933 100644 --- a/DOCKER.md +++ b/DOCKER.md @@ -1,6 +1,8 @@ # Docker Setup Guide for Morphik Core -Morphik Core provides a streamlined Docker-based setup that includes all necessary components: the core API, PostgreSQL with pgvector, and Ollama for AI models. +Morphik Core provides a streamlined Docker-based setup that includes the core API, PostgreSQL with pgvector, Redis, and optional profiles for Ollama and the web UI. + +This guide covers local development from a cloned repository with `docker-compose.yml` and `./start-dev.sh`. The hosted installer and pre-built image path use `docker-compose.run.yml` and start from the Docker-specific `morphik.docker.toml` template. ## Prerequisites @@ -16,60 +18,112 @@ git clone https://github.com/morphik-org/morphik-core.git cd morphik-core ``` -2. First-time setup: +2. Create the required Docker Compose `.env` file before first start: + + ```bash + cp .env.example .env + ``` + + Telemetry is enabled by default for self-hosted deployments. To opt out before any service starts, add this line to `.env` before continuing: + + ```dotenv + TELEMETRY=false + ``` + + You can add other environment overrides to the same file; see [Environment Variables](#3-environment-variables). + +3. Choose a Docker-reachable model configuration before first start. + + The repository's checked-in `morphik.toml` is a local development config. It enables auth bypass and selects Ollama models at `http://localhost:11434`, which is useful for direct local runs but not reachable from inside Docker containers. Pick one Docker path before starting services: + + - For an OpenAI-backed Docker setup, copy the Docker template into `morphik.toml` if you do not have local edits, then set `OPENAI_API_KEY` in `.env`. The Docker template selects OpenAI models for model-backed ingestion, parsing, and query operations. + + ```bash + cp morphik.docker.toml morphik.toml + ``` + + - For Ollama inside Docker Compose, keep or select Ollama-backed models in `morphik.toml`, update the selected Ollama `api_base` values to `http://ollama:11434`, and start with the `ollama` profile. For the checked-in `morphik.toml`, this updates the Docker-facing Ollama endpoints: + + ```bash + python - <<'PY' + from pathlib import Path + + path = Path("morphik.toml") + text = path.read_text() + text = text.replace('api_base = "http://localhost:11434"', 'api_base = "http://ollama:11434"') + path.write_text(text) + PY + ``` + + The included Ollama entrypoint pulls `nomic-embed-text` and `llama3.2`. The checked-in config also selects `qwen2.5vl:latest` for vision-capable completion and parsing, so pull that model after the Ollama service starts, or change those `morphik.toml` selections to a model that is already pulled: + + ```bash + docker compose exec ollama ollama pull qwen2.5vl:latest + ``` + +4. First-time setup: ```bash -docker compose up --build +./start-dev.sh --build ``` This command will: - Build all required containers -- Download necessary AI models (nomic-embed-text and llama3.2) - Initialize the PostgreSQL database with pgvector -- Start all services +- Start the API, worker, Redis, and PostgreSQL services + +If you chose the Ollama-in-Compose path, include the optional Ollama profile: + +```bash +COMPOSE_PROFILES=ollama ./start-dev.sh --build +``` -The initial setup may take 5-10 minutes depending on your internet speed, as it needs to download the AI models. +When the Ollama profile is enabled in the development compose file, the Ollama entrypoint pulls `nomic-embed-text` and `llama3.2`; this can add several minutes to the first startup depending on your internet speed. -3. For subsequent runs: +5. For subsequent runs: ```bash -docker compose up # Start all services -docker compose down # Stop all services +./start-dev.sh # Start services with the port from morphik.toml +docker compose down # Stop services ``` -4. To completely reset (will delete all data and models): +6. To completely reset (will delete all data and models): ```bash docker compose down -v ``` -> **Note:** If you enabled the optional UI profile (or any other compose profile), make sure to include `--profile ui` when stopping services (`docker compose --profile ui down --volumes --remove-orphans`). The hosted installer generates a `stop-morphik` script that does this for you automatically. +> **Note:** If you enabled compose profiles, include the same profiles when stopping services, for example `docker compose --profile ollama down` or `docker compose --profile ui --profile ollama down --volumes --remove-orphans`. The hosted installer generates a `stop-morphik` script that does this for you automatically. ## Configuration ### 1. Default Setup -The default configuration works out of the box and includes: +The development compose stack includes: - PostgreSQL with pgvector for document storage -- Ollama for AI models (embeddings and completions) +- Model selection through `registered_models` in `morphik.toml` - Local file storage -- Basic authentication +- Development auth bypass when using the repository's checked-in `morphik.toml` + +Model-backed operations require the selected provider to be reachable from the containers. For the Docker template's OpenAI selections, set `OPENAI_API_KEY` in `.env`. For local models, use a Docker-reachable endpoint such as `http://ollama:11434` when running Ollama as a compose profile. -### 2. Configuration File (morphik.toml) +### 2. Configuration File (`morphik.toml`) -The default `morphik.toml` is configured for Docker and includes: +Docker Compose mounts the host `./morphik.toml` file into the API and worker containers. The installer and published Docker image use the Docker-specific `morphik.docker.toml` template as the starting point, then expose it as `morphik.toml` for local edits. Model providers are configured under `registered_models`, then selected by name: ```toml [api] host = "0.0.0.0" # Important: Use 0.0.0.0 for Docker port = 8000 +[registered_models] +openai_gpt4-1-mini = { model_name = "gpt-4.1-mini" } +openai_embedding = { model_name = "text-embedding-3-small" } + [completion] -provider = "ollama" -model_name = "llama3.2" -base_url = "http://ollama:11434" # Use Docker service name +model = "openai_gpt4-1-mini" # Reference to a key in registered_models [embedding] -provider = "ollama" -model_name = "nomic-embed-text" -base_url = "http://ollama:11434" # Use Docker service name +model = "openai_embedding" # Reference to a key in registered_models +dimensions = 1536 +similarity_metric = "cosine" [database] provider = "postgres" @@ -84,16 +138,27 @@ storage_path = "/app/storage" ### 3. Environment Variables -Create a `.env` file to customize these settings: +Docker Compose reads settings from the required `.env` file. If you followed the Quick Start, it was created from `.env.example`; update it to customize these settings: ```bash -JWT_SECRET_KEY=your-secure-key-here # Important: Change in production -OPENAI_API_KEY=sk-... # Only if using OpenAI -HOST=0.0.0.0 # Leave as is for Docker -PORT=8000 # Change if needed +JWT_SECRET_KEY=your-secure-key-here # Important: change in production +SESSION_SECRET_KEY=your-session-key-here # Important: change in production +OPENAI_API_KEY=sk-... # Only if using OpenAI +ANTHROPIC_API_KEY= # Only if using Anthropic +GEMINI_API_KEY= # Only if using Gemini +LOCAL_URI_PASSWORD= # Optional: enables local URI generation +# TELEMETRY=false # Optional: disable telemetry before first start ``` -### 4. Custom Configuration +Telemetry is enabled by default for self-hosted deployments. Set `TELEMETRY=false` in `.env` before starting services if your deployment should opt out; see [Morphik telemetry](docs/telemetry.md). Change API host, port, model, storage, and provider-selection settings in `morphik.toml`, not in `.env`. Docker Compose injects the container `POSTGRES_URI` for the bundled PostgreSQL service. The current image startup check still expects that bundled service at host `postgres` with user/database `morphik`, so rotated database users, renamed databases, or external database endpoints require updating the compose settings and image startup check together. + +### 4. Security and Local-Only Defaults + +The `./start-dev.sh` path is intended for local development. The repository's checked-in `morphik.toml` sets `bypass_auth_mode = true`, so JWT and session secrets do not protect the API until auth bypass is disabled in `morphik.toml`. + +The bundled PostgreSQL service also uses the default `morphik/morphik` credentials and publishes port `5432` to the host. Keep this stack bound to a trusted local machine. Before any non-local deployment, remove or restrict host database port publishing; rotating the bundled database credentials or replacing the database service also requires updating the compose files and the image startup check that currently expects the bundled defaults. + +### 5. Custom Configuration To use your own configuration: 1. Create a custom `morphik.toml` @@ -145,12 +210,12 @@ services: - Ensure sufficient RAM (8GB+ recommended) - Check disk space: `df -h` -## Production Deployment +## Non-local Deployment Checklist -For production environments: +For production or other non-local environments, prefer the hosted installer or pre-built image path that uses `docker-compose.run.yml`; treat this repository-clone compose file as a development starting point. 1. **Security**: - - Change the default `JWT_SECRET_KEY` + - Change the default `JWT_SECRET_KEY` and `SESSION_SECRET_KEY` - Use proper network security groups - Enable HTTPS (recommended: use a reverse proxy) - Regularly update containers and dependencies diff --git a/README.md b/README.md index ab4734bd..f8ca9042 100644 --- a/README.md +++ b/README.md @@ -46,17 +46,19 @@ The best part? Morphik has a [free tier](https://www.morphik.ai/pricing)! Get st ## Table of Contents - [Getting Started with Morphik](#getting-started-with-morphik-recommended) -- [Self-hosting Morphik](#self-hosting-the-open-source-version) +- [Self-hosting Morphik](#self-hosting-morphik) - [Using Morphik](#using-morphik) - [Contributing](#contributing) -- [Open source vs paid](#License) +- [Open source vs paid](#license) ## Getting Started with Morphik (Recommended) The fastest and easiest way to get started with Morphik is by signing up for free at [Morphik](https://www.morphik.ai/signup). We have a generous free tier and transparent, compute-usage based pricing if you're looking to ingest a lot of data. ## Self-hosting Morphik -If you'd like to self-host Morphik, you can find the dedicated instruction [here](https://morphik.ai/docs/getting-started). We offer options for direct installation and installation via docker. +Self-hosted Morphik Core enables telemetry by default; set `TELEMETRY=false` before first start to opt out. See [Morphik telemetry](docs/telemetry.md) for recorded fields and upload destinations. + +If you'd like to self-host Morphik, you can find the dedicated instruction [here](https://morphik.ai/docs/getting-started). We offer options for direct installation and installation via docker. The repository-local Docker development workflow is also documented in [DOCKER.md](DOCKER.md). **Important**: Due to limited resources, we cannot provide full support for self-hosted deployments. We have an installation guide, and a [Discord community](https://discord.gg/BwMtv3Zaju) to help, but we can't guarantee full support. diff --git a/docs/telemetry.md b/docs/telemetry.md index f90e7bf7..cc8ba005 100644 --- a/docs/telemetry.md +++ b/docs/telemetry.md @@ -1,5 +1,69 @@ # Morphik Telemetry -Morphik logs minimal operational metadata (operation name, status, duration, token counts) to `logs/telemetry/` so we can keep deployments healthy, then periodically uploads those JSONL files to `https://logs.morphik.ai` to avoid unbounded disk usage. +Morphik Core includes operational telemetry for self-hosted deployments. Telemetry is enabled by default. -Telemetry is enabled by default; set `TELEMETRY=false` in the environment if you need to disable it locally, and contact founders@morphik.ai for additional compliance questions. +## Quick Summary + +- Telemetry is enabled by default for self-hosted deployments. +- When enabled, events are written locally and uploaded periodically by default; heartbeats are sent separately to `https://logs.morphik.ai`. +- To opt out, set `TELEMETRY=false` before starting Morphik services. +- Each API process startup can send an initial `first_start` heartbeat with its current `installation_id`; after the first successful heartbeat, later pings from that process are labeled `heartbeat`. Setting `TELEMETRY=false` and restarting services stops future telemetry, but it does not retract data already sent. +- Recorded data can include an installation identifier, selected metadata, and raw error text. + +## Disable Telemetry + +Set `TELEMETRY=false` in the environment for every Morphik process that should not emit telemetry. For Docker-based self-hosting, put the setting in the `.env` file used by Docker Compose before starting services: + +```dotenv +TELEMETRY=false +``` + +For direct local runs, export the variable before starting the API or worker: + +```bash +export TELEMETRY=false +``` + +`TELEMETRY=false` is the opt-out switch. The `[telemetry]` TOML section controls telemetry labels and uploader behavior, but it does not disable telemetry by itself. + +If telemetry is enabled when an API process starts, Morphik can send an initial heartbeat labeled `first_start` immediately after startup. This label is per API process lifecycle, not a one-time installation marker; failed or rejected attempts can retry with the `first_start` label until one succeeds. Set `TELEMETRY=false` before the first API or worker process starts when the deployment should avoid sending telemetry from the beginning. + +## What Is Recorded + +When enabled, Morphik writes JSONL event files under `logs/telemetry/`. Event records can include operation type, status, duration, token counts, `installation_id`, `user_id`, `app_id`, trace identifiers, worker process ID, error messages for failed operations, and selected metadata. + +Morphik reads or creates the installation identifier at `~/.databridge/installation_id`. It persists only as long as that path is preserved. The Docker compose files mount `storage`, `logs`, and `morphik.toml`, but they do not mount `~/.databridge`, so container recreation can rotate the identifier unless that path is mounted separately. + +Metadata sanitization is exact-key based, not semantic. Metadata keys named `metadata`, `request_dump`, and `request_body` are dropped. Metadata keys named `query`, `folder_name`, `folder_path`, and `full_path` are redacted. Equivalent values may still be emitted when captured under other scalar keys, including filenames, document IDs, chat IDs, end-user IDs, folder names or paths under keys such as `name` or `folder_id`, and folder descriptions under `description`. String metadata values are truncated to 256 characters, and nested list/dict metadata values are dropped. + +Failed operations may include raw exception text in the event `error` field. Error text is not covered by the metadata key redaction list or the 256-character metadata truncation rule, so provider errors, paths, IDs, or other request-derived details can appear there. + +## Where It Goes + +The API startup process starts a heartbeat job and a telemetry log uploader when telemetry is enabled. + +- Heartbeats are sent to `https://logs.morphik.ai/api/heartbeat` with fields including `project_name`, `installation_id`, timestamp, version, event type, and signature. +- Telemetry uploads are sent to `https://logs.morphik.ai/api/events/upload`. The upload envelope includes fields such as `installationId`, `startedAt`, `finishedAt`, `eventCount`, `uploaderVersion`, `workerPids`, service/environment/project metadata, byte counts, signature, and a base64-encoded gzip payload containing the JSONL events. +- After a successful upload, Morphik truncates the uploaded `logs/telemetry/usage_events_worker_*.jsonl` files. +- The `/logs` API returns an empty response when the authenticated context has no `app_id`; the checked-in development `morphik.toml` uses auth bypass, so `/logs` stays empty there until auth bypass is disabled and requests carry a real app scope. Requests up to four hours read only local files filtered by both `user_id` and `app_id`, so they can omit recent events that were already uploaded and truncated. Requests over four hours query only `https://logs.morphik.ai/api/events/query` for uploaded events filtered by `app_id`; they do not merge local files, so larger windows can include other users' uploaded events for the same app and can omit recent events that have not uploaded yet. + +## Configuration + +Telemetry-related keys in the root `morphik.toml` sample configuration include: + +```toml +[telemetry] +service_name = "databridge-core" +project_name = "" +upload_interval_hours = 4.0 +max_local_bytes = 1073741824 +``` + +- `service_name`: service label used in telemetry resource and upload metadata. +- `project_name`: optional project label used by heartbeat and upload metadata. Empty values fall back to the default OSS project label. The Docker-specific `morphik.docker.toml` template sets this to `oss_docker`; the installer and published Docker image expose that template as `morphik.toml`. +- `upload_interval_hours`: how often the log uploader attempts to send local telemetry files. +- `max_local_bytes`: total byte budget enforced across the `logs/` directory after successful uploads. If the directory exceeds this budget, the uploader removes the oldest files under `logs/`, not only files under `logs/telemetry/`. This budget is not enforced continuously; if uploads are disabled with `upload_interval_hours = 0` or uploads keep failing, local telemetry files can continue to grow. + +Set `upload_interval_hours = 0` only if you want to stop the log uploader while leaving local telemetry and heartbeat behavior enabled. Use `TELEMETRY=false` when the deployment should opt out of telemetry. + +For additional compliance questions, contact founders@morphik.ai.