Skip to content

README.md

Latest commit

 

History

History
174 lines (129 loc) · 13.5 KB

README.md

File metadata and controls

174 lines (129 loc) · 13.5 KB
Raw

AutoMem scripts

Operational, migration, recovery, and evaluation tooling for an AutoMem instance. This is the canonical inventory: every active executable in scripts/, when to use it, and the safest way to start.

Run commands from the repository root with the project environment available:

source .venv/bin/activate

Most CLI scripts support --help; cleanup_memory_types.py and recover_from_qdrant.py intentionally have no flags, so use their documented commands only. For narrative runbooks, follow the linked documents rather than inventing a new sequence.

How to read this

Most Python scripts are self-documenting: run python scripts/<name>.py --help, or read the docstring at the top of the file. Most connect to FalkorDB (and Qdrant) using credentials from .env in the repo root or ~/.config/automem/.env. A handful of harness scripts are fronted by make targets (noted inline).

Each script is tagged by lifecycle — the single most important thing to know before running it:

Tag Meaning
read-only Inspects data or streams events without writing.
maintenance Operational tool that may write or queue work; review the scope and back up first when it changes stored data.
one-time Run once per instance or per upgrade. Idempotent where noted, but not part of day-to-day ops. See docs/MIGRATIONS.md.
recovery Break-glass and potentially destructive. Only after data loss or corruption. See docs/MONITORING_AND_BACKUPS.md.
dev Local development / deployment helpers.
bench · lab Contributor evaluation and recall-tuning harnesses. Not needed to run AutoMem. See docs/TESTING.md and docs/RECALL_QUALITY_LAB.md.

Before running a script that can change FalkorDB or Qdrant, confirm the target from .env, take a backup, and start with --dry-run or a small --limit when the script offers one. Do not run recovery, re-embedding, or migration scripts against production while writers are active.

Operations and maintenance

Use these for normal operations or deliberate small maintenance tasks.

Script Lifecycle When to use it Start here
backup_automem.py maintenance Create a portable FalkorDB + Qdrant backup before a migration or on a schedule. python scripts/backup_automem.py; add --s3-bucket … --cleanup --keep 7 for off-site retention.
health_monitor.py maintenance Continuously check API, FalkorDB, Qdrant, and drift; use alert-only mode by default. python scripts/health_monitor.py --once or --interval 300. --auto-recover is an explicit, high-risk opt-in. See docs/HEALTH_MONITORING.md.
automem_watch.py read-only Observe live store/recall/update/delete, enrichment, and consolidation events. python scripts/automem_watch.py --url "$AUTOMEM_API_URL" --token "$AUTOMEM_API_TOKEN"
audit_relevance.py read-only Inspect relevance_score distribution before or after scoring changes. python scripts/audit_relevance.py reads the newest backup; use --live only for the configured instance.
reembed_embeddings.py maintenance Replace vectors after an embedding provider, model, or dimension migration. After backing up and recreating the collection, run python scripts/reembed_embeddings.py --batch-size 32. Requires QDRANT_URL; use --limit 100 only for a smoke check. See docs/MIGRATIONS.md.
reclassify_with_llm.py maintenance Reclassify fallback type='Memory' records after changing classification logic. Start with python scripts/reclassify_with_llm.py --dry-run --limit 25; apply only after review (the script asks for confirmation unless --yes is supplied).
reenrich_batch.py maintenance Queue a small batch for re-enrichment after an enrichment logic change. python scripts/reenrich_batch.py --limit 10; this calls the configured API and queues work.

browse_memories.py — read-only database browser

Interactive CLI over the production FalkorDB graph + Qdrant vectors. Connects with .env credentials and never modifies data. Four subcommands:

# search — by text, date range, type, tag, importance
python scripts/browse_memories.py search --text "Eva" --from 2025-10
python scripts/browse_memories.py search --type Decision --min-importance 0.8 --sort relevance -n 50
python scripts/browse_memories.py search --text "old project" --include-archived

# inspect — full record for one memory (4+ char id prefix works)
python scripts/browse_memories.py inspect 2751e70e

# stats — overview; --full adds a FalkorDB↔Qdrant consistency check
python scripts/browse_memories.py stats --full

# diagnose — why a memory isn't surfacing in recall (decay, access,
# relationships, importance floor, embedding quality, current weights)
python scripts/browse_memories.py diagnose 2751e70e

inspect shows full content, all FalkorDB properties, Qdrant presence/payload, and every graph relationship. diagnose reports issues at [CRITICAL] / [WARNING] / [INFO] severity.

One-time migrations

Run when adopting a specific upgrade or repair. Most are idempotent, but they are not routine operations; back up first and follow the linked runbook.

Script When to use it Start here
migrate_mcp_sqlite.py Moving from the legacy MCP sqlite_vec.db store into AutoMem. Preview first: python scripts/migrate_mcp_sqlite.py --dry-run; then pass --db, --automem-url, and --api-token.
migrate_entity_nodes.py Adopting first-class Entity nodes for legacy entity:{category}:{slug} tags. python scripts/migrate_entity_nodes.py --dry-run, then rerun without the flag.
backfill_tag_prefixes.py Restoring or introducing tag_prefixes so prefix recall remains consistent. python scripts/backfill_tag_prefixes.py --dry-run; apply after review. Use --no-qdrant only when vector payload sync is intentionally deferred.
rescore_relevance.py Repairing relevance scores produced with the previous over-aggressive decay formula. python scripts/rescore_relevance.py --dry-run, then choose the intended --target before applying.
cleanup_memory_types.py Repairing legacy invalid types such as session_start or interaction. Back up, verify .env targets the intended instance, then run python scripts/cleanup_memory_types.py. It has no preview mode.

scripts/lab/repair_entity_tags.py (lab, below) is the companion repair tool for entity-tag noise on a local clone before promoting entity nodes.

Break-glass recovery

Only reach for these after data loss or corruption. See docs/MONITORING_AND_BACKUPS.md.

Script When to use it Start here
restore_from_backup.py Restoring one or both stores from a tested local backup or API-exported tarball. Always preview first: python scripts/restore_from_backup.py --dry-run. Then select --backup-timestamp or --backup-dir, optionally --falkordb-only / --qdrant-only, and use --force only after review.
recover_from_qdrant.py FalkorDB is lost or corrupt while Qdrant is known-good and complete. Destructive: it clears the configured FalkorDB graph, then rebuilds it from Qdrant. Back up and verify Qdrant before python scripts/recover_from_qdrant.py.
deduplicate_qdrant.py Qdrant contains duplicates, commonly after a failed recovery or manual import. python scripts/deduplicate_qdrant.py --dry-run; review, then rerun with --yes to delete.

Developer and deployment

Script When to use it Start here Make target
bootstrap_dev.sh Setting up or repairing a local contributor environment. make install (or ./scripts/bootstrap_dev.sh) creates .venv, refreshes venv -> .venv, installs development dependencies, and installs pre-commit hooks. make install
deploy_check.sh Checking that Railway is deploying the expected Git commit. ./scripts/deploy_check.sh automem; use DEPLOY_CHECK_QUIET=1 only for CI-style exit-code checks. Requires linked Railway and GitHub CLIs. make deploy-check

Benchmark harness — scripts/bench/

Snapshot-based LoCoMo / LongMemEval evaluation. See docs/TESTING.md.

Script When to use it Start here Make target
bench/ingest_and_snapshot.sh Creating a fresh benchmark snapshot after a corpus or embedding change. make bench-ingest BENCH=locomo; this starts Docker and writes reusable snapshots. make bench-ingest BENCH=locomo
bench/restore_and_eval.sh Evaluating one scoring configuration against an existing snapshot. make bench-eval BENCH=locomo CONFIG=baseline make bench-eval
bench/compare_configs.sh A/B comparing two scoring configurations on the same snapshot. make bench-compare BENCH=locomo BASELINE=baseline CONFIG=<candidate> make bench-compare
bench/compare_branch.sh Comparing a branch against main using a snapshot. make bench-compare-branch BRANCH=<branch>; the script temporarily checks out refs, so begin from a clean worktree. make bench-compare-branch
bench/compare_results.py Reading two existing result JSON files without rerunning a benchmark. python scripts/bench/compare_results.py --baseline <base.json> --test <candidate.json> —
bench/analyze_locomo_results.py Producing a Markdown failure report from a LoCoMo result JSON. python scripts/bench/analyze_locomo_results.py <results.json> --output report.md —
bench/health_check.py Checking post-restore score distributions, entity quality, latency, and curated-query precision. make bench-health make bench-health
run_longmemeval_watch.sh Running LongMemEval with persistent logging and local completion/crash notifications. make test-longmemeval-watch; for a smaller run use ./scripts/run_longmemeval_watch.sh --max-questions 50. make test-longmemeval-watch

Recall Quality Lab — scripts/lab/

Data-driven recall scoring experiments against a clone of production. Full workflow: docs/RECALL_QUALITY_LAB.md.

Script When to use it Start here Make target
lab/clone_production.sh Creating an isolated local copy of production data for recall experiments. make lab-clone; use --restore-only <snapshot> for repeat experiments so production is not contacted again. make lab-clone
lab/create_test_queries.py Generating a natural-language evaluation set from a local clone. make lab-queries or python scripts/lab/create_test_queries.py --count 100. make lab-queries
lab/run_recall_test.py Measuring one configuration, an A/B comparison, or a parameter sweep. make lab-test CONFIG=baseline; use make lab-compare or make lab-sweep for the other modes. make lab-test · make lab-compare · make lab-sweep
lab/repair_entity_tags.py Repairing noisy generated entity tags on a local clone. Plan first: python scripts/lab/repair_entity_tags.py --mode canonicalize-safe; review <report-dir>/plan.jsonl, then apply with --execute --plan <report-dir>/plan.jsonl or undo with --rollback <report-dir>/rollback.jsonl. —
lab/lab_metrics.py Library module, not a CLI; implements deterministic Recall@K, MRR, NDCG, and distractor-rate metrics. Import from run_recall_test.py or tests. —
lab/lab_corpus.py Library module, not a CLI; centralizes injectable recall/corpus HTTP helpers. Import from run_recall_test.py or tests. —
lab/configs/ Creating named JSON scoring-weight overrides for A/B tests. Copy baseline.json, edit weights, then pass the filename without .json to CONFIG. —

Shared

File What it is
lib/common.sh Support module, not a CLI. Provides color helpers and wait_for_api to benchmark shell scripts.
Dockerfile.health-monitor Container recipe for running health_monitor.py in alert-only mode. Use only when you intentionally operate monitoring as a separate service.

See also