Skip to content

docs(storage): document CRAWLEE_STORAGE_DIR for local file-backed storage #2738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
DaveHanns wants to merge 1 commit into
base: master
Choose a base branch
from
+15 −5
Draft

docs(storage): document CRAWLEE_STORAGE_DIR for local file-backed storage #2738

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions sources/platform/storage/dataset.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,10 +192,10 @@ Additionally the SDK offers other methods like [`getData()`](/sdk/js/reference/c
If you have chosen to store your dataset locally, you can find it in the location below.

```text
{APIFY_LOCAL_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json
{CRAWLEE_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json
```

`DATASET_ID` refers to the dataset's _name_ or _ID_. The default dataset will be stored in the _default_ directory.
`DATASET_ID` refers to the dataset's _name_ or _ID_. The default dataset will be stored in the _default_ directory. See [Key-value store](/platform/storage/key-value-store#javascript-sdk) for the `CRAWLEE_STORAGE_DIR` environment variable that controls this path when you run the Actor outside `apify run`.

To add data to the default dataset, you can use the example below:

Expand Down
12 changes: 11 additions & 1 deletion sources/platform/storage/key_value_store.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,11 +151,21 @@
You can find _INPUT.json_ and other key-value store files in the location below.

```text
{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}
{CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}
```

The default key-value store's ID is _default_. The `{KEY}` is the record's _key_ and `{EXT}` corresponds to the record value's MIME content type.

:::info Local storage directory

When you run an Actor locally through `apify run`, the CLI sets `CRAWLEE_STORAGE_DIR=./storage` in the Actor's process. If you launch your entry point directly (for example, `tsx src/main.ts` or `node dist/main.js`), you need to set the variable yourself — otherwise the SDK falls back to the platform client and fails with `ApifyApiError: Authentication token was not provided` when it tries to reach `api.apify.com`. Setting `APIFY_IS_AT_HOME=0` on its own is not enough; the SDK still needs `CRAWLEE_STORAGE_DIR` to know where the file-backed storage lives.

Check failure on line 161 in sources/platform/storage/key_value_store.md

View workflow job for this annotation

GitHub Actions / lint 

[vale] reported by reviewdog 🐶
[Microsoft.Dashes] Remove the spaces around ' — '.

Raw Output:
{"message": "[Microsoft.Dashes] Remove the spaces around ' — '.", "location": {"path": "sources/platform/storage/key_value_store.md", "range": {"start": {"line": 161, "column": 255}}}, "severity": "ERROR"}

```bash
CRAWLEE_STORAGE_DIR=./storage tsx src/main.ts
```

:::

To manage your key-value stores, you can use the following methods. See the `KeyValueStore` class's [API reference](/sdk/js/reference/class/KeyValueStore) for the full list.

```js
Expand Down
4 changes: 2 additions & 2 deletions sources/platform/storage/request_queue.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,10 +159,10 @@ Every Actor run is automatically linked with a default request queue, initiated
If you are storing your data locally, you can find your request queue at the following location.

```text
{APIFY_LOCAL_STORAGE_DIR}/request_queues/{QUEUE_ID}/{ID}.json
{CRAWLEE_STORAGE_DIR}/request_queues/{QUEUE_ID}/{ID}.json
```

The default request queue's ID is _default_. Each request in the queue is stored as a separate JSON file, where `{ID}` is a request ID.
The default request queue's ID is _default_. Each request in the queue is stored as a separate JSON file, where `{ID}` is a request ID. See [Key-value store](/platform/storage/key-value-store#javascript-sdk) for the `CRAWLEE_STORAGE_DIR` environment variable that controls this path when you run the Actor outside `apify run`.

To open a request queue, use the [`Actor.openRequestQueue()`](/sdk/js/reference/class/Actor#openRequestQueue) method.

Expand Down
Loading