docs(storage): document CRAWLEE_STORAGE_DIR for local file-backed storage#2738
Draft
DaveHanns wants to merge 1 commit into
Draft
docs(storage): document CRAWLEE_STORAGE_DIR for local file-backed storage#2738DaveHanns wants to merge 1 commit into
DaveHanns wants to merge 1 commit into
Conversation
…rage
The JavaScript SDK sections of the storage pages referred to
`{APIFY_LOCAL_STORAGE_DIR}` as the placeholder for the local storage
root, but Apify SDK v3 no longer respects that variable — it reads
`CRAWLEE_STORAGE_DIR` (inherited from Crawlee v3). `apify run` sets it
automatically, so users relying on the CLI never notice, but users who
launch their compiled entry point directly (`tsx src/main.ts`,
`node dist/main.js`, unit tests, IDE run configurations, ...) hit:
ApifyApiError: Authentication token was not provided
path: /v2/key-value-stores
because the SDK falls back to the platform client when no local
storage directory is configured. Setting `APIFY_IS_AT_HOME=0` is not
enough on its own — the SDK still needs `CRAWLEE_STORAGE_DIR` to know
where to persist the default key-value store, dataset, and request
queue.
Update the three JS SDK sections to use `{CRAWLEE_STORAGE_DIR}` in the
path placeholder, and add an info callout in the key-value store page
with the correct env var, the failure mode, and a working example
(`CRAWLEE_STORAGE_DIR=./storage tsx src/main.ts`). Cross-link from the
dataset and request queue pages to the callout.
The Python SDK sections are left untouched — they use a different
storage backend and env var, and this finding is JS-specific.
Surfaced during an evaluation of Apify surfaces for agent-driven Actor development.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
✅ Preview for this PR (commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The JavaScript SDK sections of the storage pages refer to
{APIFY_LOCAL_STORAGE_DIR}as the placeholder for the local storage root. That variable is no longer respected by Apify SDK v3 — it readsCRAWLEE_STORAGE_DIR(inherited from Crawlee v3).apify runsets it automatically (seesrc/commands/run.tsinapify-cli), so users staying on the CLI never see the mismatch. But users who launch their compiled entry point directly (tsx src/main.ts,node dist/main.js, unit tests, IDE run configurations, ...) get bitten.Reproducer
Expected: the SDK persists the default key-value store / dataset / request queue under
./storage.Actual:
The SDK ignores
APIFY_LOCAL_STORAGE_DIR, treats storage as unconfigured, and falls back to the platform HTTP client — which then fails because noAPIFY_TOKENwas provided. SettingAPIFY_IS_AT_HOME=0on its own is not enough. The fix is:Nothing in the docs points to
CRAWLEE_STORAGE_DIR, so the only way to discover it today is to grepnode_modules/@apify/*ornode_modules/@crawlee/*.Changes
sources/platform/storage/key_value_store.md— JS SDK section: rename the path placeholder to{CRAWLEE_STORAGE_DIR}and add an info callout explaining the env var, the failure mode, and a working example.sources/platform/storage/dataset.md— JS SDK section: rename the path placeholder and cross-link to the callout.sources/platform/storage/request_queue.md— JS SDK section: same.The Python SDK sections are left untouched — the Python SDK uses a different storage backend and env var, and this issue is JS-specific.
Test plan
pnpm build(or the docs preview) renders the three pages without broken links or admonition errors.CRAWLEE_STORAGE_DIR=./storage tsx src/main.tsexample on a fresh v3 template and confirm files land under./storage/key_value_stores/default/.Surfaced during an evaluation of Apify surfaces for agent-driven Actor development.