fix: filter sitemap-derived URLs by enqueue strategy

[vdusek]

SitemapRequestList, Sitemap.load / parseSitemap, and RobotsTxtFile.getSitemaps() now apply an enqueue strategy to the URLs they extract. They keep only entries that match the strategy (default same-hostname) relative to their parent sitemap, and always drop non-http(s) schemes. This brings sitemap loading in line with enqueueLinks, which already scopes discovered links to the same hostname by default.

The selected strategy is also stamped onto the emitted Request objects, so it keeps being enforced after navigation (e.g. across redirects).

This changes the default behavior: cross-host sitemap entries are no longer enqueued unless you opt in with enqueueStrategy: 'all' (or 'same-domain' / 'same-origin').

[B4nan]

same here

Suggested change

	export type EnqueueStrategyValue = 'all' | 'same-hostname' | 'same-domain' | 'same-origin';
	export type EnqueueStrategyValue = `${EnqueueStrategy}`;

i would rather inline this instead of introducing a new exported type for it

[vdusek]

This is not possible because crawlee/core already depends on crawlee/utils, so that would be a circular reference. Am I correct? 🙂

[B4nan]

type-level cycles are fine

[vdusek]

I won't pretend to be a guru here, but this still doesn't seem doable (or at least not doable in a right & easy way):

Type-level cycles are fine within a single tsc program (or with project references) — but that's not the situation here. @crawlee/utils and @crawlee/core are two separately-compiled packages, and this repo doesn't use project references: each package is built by its own tsc -p tsconfig.build.json, ordered topologically by Turbo from the package.json dependency graph.

To write ${EnqueueStrategy} in utils I'd have to import type { EnqueueStrategy } from '@crawlee/core', and that hits two concrete walls:

1. It requires @crawlee/core in utils' package.json. As soon as I add it, Turbo refuses to build: x Cyclic dependency detected: @crawlee/core#build, @crawlee/utils#build
2. Even without declaring it (relying on the workspace symlink), the build fails. tsconfig.build.json has no paths mapping — only the root dev/type-check tsconfig.json does — so cross-package types resolve from the emitted node_modules/@crawlee/core/dist/index.d.ts. That file only exists after core is built, but core builds after utils (core depends on utils), so on a clean build utils compiles first and gets TS2307: Cannot find module '@crawlee/core'.

So I kept the mirrored literal union in utils. I did apply the ${EnqueueStrategy} form in sitemap_request_list.ts, since that's in core where the enum is already in scope. If we ever want to genuinely dedupe it, the value union would have to move into @crawlee/types (which both packages depend on), but that's a bigger change than this nit.

[vdusek]

@B4nan FYI; this is still a draft - I haven't reviewed most of it yet

[B4nan]

no worries, i was just curious 🙃

[vdusek]

Requesting a review from @janbuchar, since you may have additional context from the Python solution.

[janbuchar]

LGTM, thanks