From 1ba32edafbcc0664b1b88268d38292c4dd274401 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20Hanu=C5=A1?= Date: Sun, 5 Jul 2026 01:06:39 +0200 Subject: [PATCH] docs(storage): document CRAWLEE_STORAGE_DIR for local file-backed storage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The JavaScript SDK sections of the storage pages referred to `{APIFY_LOCAL_STORAGE_DIR}` as the placeholder for the local storage root, but Apify SDK v3 no longer respects that variable — it reads `CRAWLEE_STORAGE_DIR` (inherited from Crawlee v3). `apify run` sets it automatically, so users relying on the CLI never notice, but users who launch their compiled entry point directly (`tsx src/main.ts`, `node dist/main.js`, unit tests, IDE run configurations, ...) hit: ApifyApiError: Authentication token was not provided path: /v2/key-value-stores because the SDK falls back to the platform client when no local storage directory is configured. Setting `APIFY_IS_AT_HOME=0` is not enough on its own — the SDK still needs `CRAWLEE_STORAGE_DIR` to know where to persist the default key-value store, dataset, and request queue. Update the three JS SDK sections to use `{CRAWLEE_STORAGE_DIR}` in the path placeholder, and add an info callout in the key-value store page with the correct env var, the failure mode, and a working example (`CRAWLEE_STORAGE_DIR=./storage tsx src/main.ts`). Cross-link from the dataset and request queue pages to the callout. The Python SDK sections are left untouched — they use a different storage backend and env var, and this finding is JS-specific. Surfaced during an evaluation of Apify surfaces for agent-driven Actor development. Co-Authored-By: Claude Opus 4.7 --- sources/platform/storage/dataset.md | 4 ++-- sources/platform/storage/key_value_store.md | 12 +++++++++++- sources/platform/storage/request_queue.md | 4 ++-- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/sources/platform/storage/dataset.md b/sources/platform/storage/dataset.md index bb333cd448..d091810d2a 100644 --- a/sources/platform/storage/dataset.md +++ b/sources/platform/storage/dataset.md @@ -192,10 +192,10 @@ Additionally the SDK offers other methods like [`getData()`](/sdk/js/reference/c If you have chosen to store your dataset locally, you can find it in the location below. ```text -{APIFY_LOCAL_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json +{CRAWLEE_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json ``` -`DATASET_ID` refers to the dataset's _name_ or _ID_. The default dataset will be stored in the _default_ directory. +`DATASET_ID` refers to the dataset's _name_ or _ID_. The default dataset will be stored in the _default_ directory. See [Key-value store](/platform/storage/key-value-store#javascript-sdk) for the `CRAWLEE_STORAGE_DIR` environment variable that controls this path when you run the Actor outside `apify run`. To add data to the default dataset, you can use the example below: diff --git a/sources/platform/storage/key_value_store.md b/sources/platform/storage/key_value_store.md index 46689d6d86..ca9ec17705 100644 --- a/sources/platform/storage/key_value_store.md +++ b/sources/platform/storage/key_value_store.md @@ -151,11 +151,21 @@ Every Actor run is linked to a default key-value store that is automatically cre You can find _INPUT.json_ and other key-value store files in the location below. ```text -{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT} +{CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT} ``` The default key-value store's ID is _default_. The `{KEY}` is the record's _key_ and `{EXT}` corresponds to the record value's MIME content type. +:::info Local storage directory + +When you run an Actor locally through `apify run`, the CLI sets `CRAWLEE_STORAGE_DIR=./storage` in the Actor's process. If you launch your entry point directly (for example, `tsx src/main.ts` or `node dist/main.js`), you need to set the variable yourself — otherwise the SDK falls back to the platform client and fails with `ApifyApiError: Authentication token was not provided` when it tries to reach `api.apify.com`. Setting `APIFY_IS_AT_HOME=0` on its own is not enough; the SDK still needs `CRAWLEE_STORAGE_DIR` to know where the file-backed storage lives. + +```bash +CRAWLEE_STORAGE_DIR=./storage tsx src/main.ts +``` + +::: + To manage your key-value stores, you can use the following methods. See the `KeyValueStore` class's [API reference](/sdk/js/reference/class/KeyValueStore) for the full list. ```js diff --git a/sources/platform/storage/request_queue.md b/sources/platform/storage/request_queue.md index 9f0208abcf..6e59b39499 100644 --- a/sources/platform/storage/request_queue.md +++ b/sources/platform/storage/request_queue.md @@ -159,10 +159,10 @@ Every Actor run is automatically linked with a default request queue, initiated If you are storing your data locally, you can find your request queue at the following location. ```text -{APIFY_LOCAL_STORAGE_DIR}/request_queues/{QUEUE_ID}/{ID}.json +{CRAWLEE_STORAGE_DIR}/request_queues/{QUEUE_ID}/{ID}.json ``` -The default request queue's ID is _default_. Each request in the queue is stored as a separate JSON file, where `{ID}` is a request ID. +The default request queue's ID is _default_. Each request in the queue is stored as a separate JSON file, where `{ID}` is a request ID. See [Key-value store](/platform/storage/key-value-store#javascript-sdk) for the `CRAWLEE_STORAGE_DIR` environment variable that controls this path when you run the Actor outside `apify run`. To open a request queue, use the [`Actor.openRequestQueue()`](/sdk/js/reference/class/Actor#openRequestQueue) method.