Skip to content

docs(storage): document CRAWLEE_STORAGE_DIR for local file-backed storage#2738

Draft
DaveHanns wants to merge 1 commit into
masterfrom
docs/crawlee-storage-dir-local
Draft

docs(storage): document CRAWLEE_STORAGE_DIR for local file-backed storage#2738
DaveHanns wants to merge 1 commit into
masterfrom
docs/crawlee-storage-dir-local

Conversation

@DaveHanns

Copy link
Copy Markdown
Contributor

Summary

The JavaScript SDK sections of the storage pages refer to {APIFY_LOCAL_STORAGE_DIR} as the placeholder for the local storage root. That variable is no longer respected by Apify SDK v3 — it reads CRAWLEE_STORAGE_DIR (inherited from Crawlee v3). apify run sets it automatically (see src/commands/run.ts in apify-cli), so users staying on the CLI never see the mismatch. But users who launch their compiled entry point directly (tsx src/main.ts, node dist/main.js, unit tests, IDE run configurations, ...) get bitten.

Reproducer

# Fresh Actor scaffolded from a TypeScript template
apify create my-actor -t js-crawlee-cheerio-ts  # or any v3-based template
cd my-actor

# Run the compiled entry point directly, bypassing `apify run`
APIFY_IS_AT_HOME=0 APIFY_LOCAL_STORAGE_DIR=./storage tsx src/main.ts

Expected: the SDK persists the default key-value store / dataset / request queue under ./storage.

Actual:

ApifyApiError: Authentication token was not provided
    path: /v2/key-value-stores

The SDK ignores APIFY_LOCAL_STORAGE_DIR, treats storage as unconfigured, and falls back to the platform HTTP client — which then fails because no APIFY_TOKEN was provided. Setting APIFY_IS_AT_HOME=0 on its own is not enough. The fix is:

CRAWLEE_STORAGE_DIR=./storage tsx src/main.ts

Nothing in the docs points to CRAWLEE_STORAGE_DIR, so the only way to discover it today is to grep node_modules/@apify/* or node_modules/@crawlee/*.

Changes

  • sources/platform/storage/key_value_store.md — JS SDK section: rename the path placeholder to {CRAWLEE_STORAGE_DIR} and add an info callout explaining the env var, the failure mode, and a working example.
  • sources/platform/storage/dataset.md — JS SDK section: rename the path placeholder and cross-link to the callout.
  • sources/platform/storage/request_queue.md — JS SDK section: same.

The Python SDK sections are left untouched — the Python SDK uses a different storage backend and env var, and this issue is JS-specific.

Test plan

  • pnpm build (or the docs preview) renders the three pages without broken links or admonition errors.
  • Follow the CRAWLEE_STORAGE_DIR=./storage tsx src/main.ts example on a fresh v3 template and confirm files land under ./storage/key_value_stores/default/.

Surfaced during an evaluation of Apify surfaces for agent-driven Actor development.

…rage

The JavaScript SDK sections of the storage pages referred to
`{APIFY_LOCAL_STORAGE_DIR}` as the placeholder for the local storage
root, but Apify SDK v3 no longer respects that variable — it reads
`CRAWLEE_STORAGE_DIR` (inherited from Crawlee v3). `apify run` sets it
automatically, so users relying on the CLI never notice, but users who
launch their compiled entry point directly (`tsx src/main.ts`,
`node dist/main.js`, unit tests, IDE run configurations, ...) hit:

    ApifyApiError: Authentication token was not provided
        path: /v2/key-value-stores

because the SDK falls back to the platform client when no local
storage directory is configured. Setting `APIFY_IS_AT_HOME=0` is not
enough on its own — the SDK still needs `CRAWLEE_STORAGE_DIR` to know
where to persist the default key-value store, dataset, and request
queue.

Update the three JS SDK sections to use `{CRAWLEE_STORAGE_DIR}` in the
path placeholder, and add an info callout in the key-value store page
with the correct env var, the failure mode, and a working example
(`CRAWLEE_STORAGE_DIR=./storage tsx src/main.ts`). Cross-link from the
dataset and request queue pages to the callout.

The Python SDK sections are left untouched — they use a different
storage backend and env var, and this finding is JS-specific.

Surfaced during an evaluation of Apify surfaces for agent-driven Actor development.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@apify-service-account

apify-service-account commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

✅ Preview for this PR (commit 1ba32ed) is ready at https://pr-2738.preview.docs.apify.com (see action run).

@DaveHanns DaveHanns marked this pull request as draft July 5, 2026 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants