docs: surface telemetry controls for self-hosted deployments

.env.example

@@ -1,6 +1,9 @@
 # Required
 JWT_SECRET_KEY="your-super-secret-key-change-in-production"
 SESSION_SECRET_KEY="your-session-secret-key-change-in-production"
+
+# Direct/local runs can set POSTGRES_URI here. Docker Compose sets the container
+# POSTGRES_URI in docker-compose.yml/docker-compose.run.yml.
 POSTGRES_URI="postgresql+asyncpg://morphik:morphik@localhost:5432/morphik"
 
 # LLM Provider API Keys (set the ones you use)
@@ -18,3 +21,6 @@ TURBOPUFFER_API_KEY=
 SENTRY_DSN=
 LOCAL_URI_PASSWORD=
 LITELLM_DUMMY_API_KEY=
+
+# Uncomment and set to false before first start to disable self-hosted telemetry.
+# TELEMETRY=false

DOCKER.md

@@ -1,6 +1,8 @@
 # Docker Setup Guide for Morphik Core
 
-Morphik Core provides a streamlined Docker-based setup that includes all necessary components: the core API, PostgreSQL with pgvector, and Ollama for AI models.
+Morphik Core provides a streamlined Docker-based setup that includes the core API, PostgreSQL with pgvector, Redis, and optional profiles for Ollama and the web UI.
+
+This guide covers local development from a cloned repository with `docker-compose.yml` and `./start-dev.sh`. The hosted installer and pre-built image path use `docker-compose.run.yml` and start from the Docker-specific `morphik.docker.toml` template.
 
 ## Prerequisites
 
@@ -16,60 +18,112 @@ git clone https://github.com/morphik-org/morphik-core.git
 cd morphik-core
 ```
 
-2. First-time setup:
+2. Create the required Docker Compose `.env` file before first start:
+
+   ```bash
+   cp .env.example .env
+   ```
+
+   Telemetry is enabled by default for self-hosted deployments. To opt out before any service starts, add this line to `.env` before continuing:
+
+   ```dotenv
+   TELEMETRY=false
+   ```
+
+   You can add other environment overrides to the same file; see [Environment Variables](#3-environment-variables).
+
+3. Choose a Docker-reachable model configuration before first start.
+
+   The repository's checked-in `morphik.toml` is a local development config. It enables auth bypass and selects Ollama models at `http://localhost:11434`, which is useful for direct local runs but not reachable from inside Docker containers. Pick one Docker path before starting services:
+
+   - For an OpenAI-backed Docker setup, copy the Docker template into `morphik.toml` if you do not have local edits, then set `OPENAI_API_KEY` in `.env`. The Docker template selects OpenAI models for model-backed ingestion, parsing, and query operations.
+
+     ```bash
+     cp morphik.docker.toml morphik.toml
+     ```
+
+   - For Ollama inside Docker Compose, keep or select Ollama-backed models in `morphik.toml`, update the selected Ollama `api_base` values to `http://ollama:11434`, and start with the `ollama` profile. For the checked-in `morphik.toml`, this updates the Docker-facing Ollama endpoints:
+
+     ```bash
+     python - <<'PY'
+     from pathlib import Path
+
+     path = Path("morphik.toml")
+     text = path.read_text()
+     text = text.replace('api_base = "http://localhost:11434"', 'api_base = "http://ollama:11434"')
+     path.write_text(text)
+     PY
+     ```
+
+     The included Ollama entrypoint pulls `nomic-embed-text` and `llama3.2`. The checked-in config also selects `qwen2.5vl:latest` for vision-capable completion and parsing, so pull that model after the Ollama service starts, or change those `morphik.toml` selections to a model that is already pulled:
+
+     ```bash
+     docker compose exec ollama ollama pull qwen2.5vl:latest
+     ```
+
+4. First-time setup:
 ```bash
-docker compose up --build
+./start-dev.sh --build
 ```
 
 This command will:
 - Build all required containers
-- Download necessary AI models (nomic-embed-text and llama3.2)
 - Initialize the PostgreSQL database with pgvector
-- Start all services
+- Start the API, worker, Redis, and PostgreSQL services
+
+If you chose the Ollama-in-Compose path, include the optional Ollama profile:
+
+```bash
+COMPOSE_PROFILES=ollama ./start-dev.sh --build
+```
 
-The initial setup may take 5-10 minutes depending on your internet speed, as it needs to download the AI models.
+When the Ollama profile is enabled in the development compose file, the Ollama entrypoint pulls `nomic-embed-text` and `llama3.2`; this can add several minutes to the first startup depending on your internet speed.
 
-3. For subsequent runs:
+5. For subsequent runs:
 ```bash
-docker compose up    # Start all services
-docker compose down  # Stop all services
+./start-dev.sh       # Start services with the port from morphik.toml
+docker compose down  # Stop services
 ```
 
-4. To completely reset (will delete all data and models):
+6. To completely reset (will delete all data and models):
 ```bash
 docker compose down -v
 ```
 
-> **Note:** If you enabled the optional UI profile (or any other compose profile), make sure to include `--profile ui` when stopping services (`docker compose --profile ui down --volumes --remove-orphans`). The hosted installer generates a `stop-morphik` script that does this for you automatically.
+> **Note:** If you enabled compose profiles, include the same profiles when stopping services, for example `docker compose --profile ollama down` or `docker compose --profile ui --profile ollama down --volumes --remove-orphans`. The hosted installer generates a `stop-morphik` script that does this for you automatically.
 
 ## Configuration
 
 ### 1. Default Setup
 
-The default configuration works out of the box and includes:
+The development compose stack includes:
 - PostgreSQL with pgvector for document storage
-- Ollama for AI models (embeddings and completions)
+- Model selection through `registered_models` in `morphik.toml`
 - Local file storage
-- Basic authentication
+- Development auth bypass when using the repository's checked-in `morphik.toml`
+
+Model-backed operations require the selected provider to be reachable from the containers. For the Docker template's OpenAI selections, set `OPENAI_API_KEY` in `.env`. For local models, use a Docker-reachable endpoint such as `http://ollama:11434` when running Ollama as a compose profile.
 
-### 2. Configuration File (morphik.toml)
+### 2. Configuration File (`morphik.toml`)
 
-The default `morphik.toml` is configured for Docker and includes:
+Docker Compose mounts the host `./morphik.toml` file into the API and worker containers. The installer and published Docker image use the Docker-specific `morphik.docker.toml` template as the starting point, then expose it as `morphik.toml` for local edits. Model providers are configured under `registered_models`, then selected by name:
 
 ```toml
 [api]
 host = "0.0.0.0"  # Important: Use 0.0.0.0 for Docker
 port = 8000
 
+[registered_models]
+openai_gpt4-1-mini = { model_name = "gpt-4.1-mini" }
+openai_embedding = { model_name = "text-embedding-3-small" }
+
 [completion]
-provider = "ollama"
-model_name = "llama3.2"
-base_url = "http://ollama:11434"  # Use Docker service name
+model = "openai_gpt4-1-mini"  # Reference to a key in registered_models
 
 [embedding]
-provider = "ollama"
-model_name = "nomic-embed-text"
-base_url = "http://ollama:11434"  # Use Docker service name
+model = "openai_embedding"  # Reference to a key in registered_models
+dimensions = 1536
+similarity_metric = "cosine"
 
 [database]
 provider = "postgres"
@@ -84,16 +138,27 @@ storage_path = "/app/storage"
 
 ### 3. Environment Variables
 
-Create a `.env` file to customize these settings:
+Docker Compose reads settings from the required `.env` file. If you followed the Quick Start, it was created from `.env.example`; update it to customize these settings:
 
 ```bash
-JWT_SECRET_KEY=your-secure-key-here  # Important: Change in production
-OPENAI_API_KEY=sk-...                # Only if using OpenAI
-HOST=0.0.0.0                         # Leave as is for Docker
-PORT=8000                            # Change if needed
+JWT_SECRET_KEY=your-secure-key-here      # Important: change in production
+SESSION_SECRET_KEY=your-session-key-here # Important: change in production
+OPENAI_API_KEY=sk-...                    # Only if using OpenAI
+ANTHROPIC_API_KEY=                       # Only if using Anthropic
+GEMINI_API_KEY=                          # Only if using Gemini
+LOCAL_URI_PASSWORD=                      # Optional: enables local URI generation
+# TELEMETRY=false                        # Optional: disable telemetry before first start
 ```
 
-### 4. Custom Configuration
+Telemetry is enabled by default for self-hosted deployments. Set `TELEMETRY=false` in `.env` before starting services if your deployment should opt out; see [Morphik telemetry](docs/telemetry.md). Change API host, port, model, storage, and provider-selection settings in `morphik.toml`, not in `.env`. Docker Compose injects the container `POSTGRES_URI` for the bundled PostgreSQL service. The current image startup check still expects that bundled service at host `postgres` with user/database `morphik`, so rotated database users, renamed databases, or external database endpoints require updating the compose settings and image startup check together.
+
+### 4. Security and Local-Only Defaults
+
+The `./start-dev.sh` path is intended for local development. The repository's checked-in `morphik.toml` sets `bypass_auth_mode = true`, so JWT and session secrets do not protect the API until auth bypass is disabled in `morphik.toml`.
+
+The bundled PostgreSQL service also uses the default `morphik/morphik` credentials and publishes port `5432` to the host. Keep this stack bound to a trusted local machine. Before any non-local deployment, remove or restrict host database port publishing; rotating the bundled database credentials or replacing the database service also requires updating the compose files and the image startup check that currently expects the bundled defaults.
+
+### 5. Custom Configuration
 
 To use your own configuration:
 1. Create a custom `morphik.toml`
@@ -145,12 +210,12 @@ services:
    - Ensure sufficient RAM (8GB+ recommended)
    - Check disk space: `df -h`
 
-## Production Deployment
+## Non-local Deployment Checklist
 
-For production environments:
+For production or other non-local environments, prefer the hosted installer or pre-built image path that uses `docker-compose.run.yml`; treat this repository-clone compose file as a development starting point.
 
 1. **Security**:
-   - Change the default `JWT_SECRET_KEY`
+   - Change the default `JWT_SECRET_KEY` and `SESSION_SECRET_KEY`
    - Use proper network security groups
    - Enable HTTPS (recommended: use a reverse proxy)
    - Regularly update containers and dependencies

README.md

@@ -46,17 +46,19 @@ The best part? Morphik has a [free tier](https://www.morphik.ai/pricing)! Get st
 
 ## Table of Contents
 - [Getting Started with Morphik](#getting-started-with-morphik-recommended)
-- [Self-hosting Morphik](#self-hosting-the-open-source-version)
+- [Self-hosting Morphik](#self-hosting-morphik)
 - [Using Morphik](#using-morphik)
 - [Contributing](#contributing)
-- [Open source vs paid](#License)
+- [Open source vs paid](#license)
 
 ## Getting Started with Morphik (Recommended)
 
 The fastest and easiest way to get started with Morphik is by signing up for free at [Morphik](https://www.morphik.ai/signup). We have a generous free tier and transparent, compute-usage based pricing if you're looking to ingest a lot of data.
 
 ## Self-hosting Morphik
-If you'd like to self-host Morphik, you can find the dedicated instruction [here](https://morphik.ai/docs/getting-started). We offer options for direct installation and installation via docker.
+Self-hosted Morphik Core enables telemetry by default; set `TELEMETRY=false` before first start to opt out. See [Morphik telemetry](docs/telemetry.md) for recorded fields and upload destinations.
+
+If you'd like to self-host Morphik, you can find the dedicated instruction [here](https://morphik.ai/docs/getting-started). We offer options for direct installation and installation via docker. The repository-local Docker development workflow is also documented in [DOCKER.md](DOCKER.md).
 
 **Important**: Due to limited resources, we cannot provide full support for self-hosted deployments. We have an installation guide, and a [Discord community](https://discord.gg/BwMtv3Zaju) to help, but we can't guarantee full support.
 

docs/telemetry.md

@@ -1,5 +1,69 @@
 # Morphik Telemetry
 
-Morphik logs minimal operational metadata (operation name, status, duration, token counts) to `logs/telemetry/` so we can keep deployments healthy, then periodically uploads those JSONL files to `https://logs.morphik.ai` to avoid unbounded disk usage.
+Morphik Core includes operational telemetry for self-hosted deployments. Telemetry is enabled by default.
 
-Telemetry is enabled by default; set `TELEMETRY=false` in the environment if you need to disable it locally, and contact founders@morphik.ai for additional compliance questions.
+## Quick Summary
+
+- Telemetry is enabled by default for self-hosted deployments.
+- When enabled, events are written locally and uploaded periodically by default; heartbeats are sent separately to `https://logs.morphik.ai`.
+- To opt out, set `TELEMETRY=false` before starting Morphik services.
+- Each API process startup can send an initial `first_start` heartbeat with its current `installation_id`; after the first successful heartbeat, later pings from that process are labeled `heartbeat`. Setting `TELEMETRY=false` and restarting services stops future telemetry, but it does not retract data already sent.
+- Recorded data can include an installation identifier, selected metadata, and raw error text.
+
+## Disable Telemetry
+
+Set `TELEMETRY=false` in the environment for every Morphik process that should not emit telemetry. For Docker-based self-hosting, put the setting in the `.env` file used by Docker Compose before starting services:
+
+```dotenv
+TELEMETRY=false
+```
+
+For direct local runs, export the variable before starting the API or worker:
+
+```bash
+export TELEMETRY=false
+```
+
+`TELEMETRY=false` is the opt-out switch. The `[telemetry]` TOML section controls telemetry labels and uploader behavior, but it does not disable telemetry by itself.
+
+If telemetry is enabled when an API process starts, Morphik can send an initial heartbeat labeled `first_start` immediately after startup. This label is per API process lifecycle, not a one-time installation marker; failed or rejected attempts can retry with the `first_start` label until one succeeds. Set `TELEMETRY=false` before the first API or worker process starts when the deployment should avoid sending telemetry from the beginning.
+
+## What Is Recorded
+
+When enabled, Morphik writes JSONL event files under `logs/telemetry/`. Event records can include operation type, status, duration, token counts, `installation_id`, `user_id`, `app_id`, trace identifiers, worker process ID, error messages for failed operations, and selected metadata.
+
+Morphik reads or creates the installation identifier at `~/.databridge/installation_id`. It persists only as long as that path is preserved. The Docker compose files mount `storage`, `logs`, and `morphik.toml`, but they do not mount `~/.databridge`, so container recreation can rotate the identifier unless that path is mounted separately.
+
+Metadata sanitization is exact-key based, not semantic. Metadata keys named `metadata`, `request_dump`, and `request_body` are dropped. Metadata keys named `query`, `folder_name`, `folder_path`, and `full_path` are redacted. Equivalent values may still be emitted when captured under other scalar keys, including filenames, document IDs, chat IDs, end-user IDs, folder names or paths under keys such as `name` or `folder_id`, and folder descriptions under `description`. String metadata values are truncated to 256 characters, and nested list/dict metadata values are dropped.
+
+Failed operations may include raw exception text in the event `error` field. Error text is not covered by the metadata key redaction list or the 256-character metadata truncation rule, so provider errors, paths, IDs, or other request-derived details can appear there.
+
+## Where It Goes
+
+The API startup process starts a heartbeat job and a telemetry log uploader when telemetry is enabled.
+
+- Heartbeats are sent to `https://logs.morphik.ai/api/heartbeat` with fields including `project_name`, `installation_id`, timestamp, version, event type, and signature.
+- Telemetry uploads are sent to `https://logs.morphik.ai/api/events/upload`. The upload envelope includes fields such as `installationId`, `startedAt`, `finishedAt`, `eventCount`, `uploaderVersion`, `workerPids`, service/environment/project metadata, byte counts, signature, and a base64-encoded gzip payload containing the JSONL events.
+- After a successful upload, Morphik truncates the uploaded `logs/telemetry/usage_events_worker_*.jsonl` files.
+- The `/logs` API returns an empty response when the authenticated context has no `app_id`; the checked-in development `morphik.toml` uses auth bypass, so `/logs` stays empty there until auth bypass is disabled and requests carry a real app scope. Requests up to four hours read only local files filtered by both `user_id` and `app_id`, so they can omit recent events that were already uploaded and truncated. Requests over four hours query only `https://logs.morphik.ai/api/events/query` for uploaded events filtered by `app_id`; they do not merge local files, so larger windows can include other users' uploaded events for the same app and can omit recent events that have not uploaded yet.
+
+## Configuration
+
+Telemetry-related keys in the root `morphik.toml` sample configuration include:
+
+```toml
+[telemetry]
+service_name = "databridge-core"
+project_name = ""
+upload_interval_hours = 4.0
+max_local_bytes = 1073741824
+```
+
+- `service_name`: service label used in telemetry resource and upload metadata.
+- `project_name`: optional project label used by heartbeat and upload metadata. Empty values fall back to the default OSS project label. The Docker-specific `morphik.docker.toml` template sets this to `oss_docker`; the installer and published Docker image expose that template as `morphik.toml`.
+- `upload_interval_hours`: how often the log uploader attempts to send local telemetry files.
+- `max_local_bytes`: total byte budget enforced across the `logs/` directory after successful uploads. If the directory exceeds this budget, the uploader removes the oldest files under `logs/`, not only files under `logs/telemetry/`. This budget is not enforced continuously; if uploads are disabled with `upload_interval_hours = 0` or uploads keep failing, local telemetry files can continue to grow.
+
+Set `upload_interval_hours = 0` only if you want to stop the log uploader while leaving local telemetry and heartbeat behavior enabled. Use `TELEMETRY=false` when the deployment should opt out of telemetry.
+
+For additional compliance questions, contact founders@morphik.ai.