fix(scrapy): stop the crawl gracefully when the Actor run is migrated or aborted

[vdusek]

Closes #1103.

The Scrapy integration ignored MIGRATING and ABORTING, so the process was killed with requests in flight. Those stay pending, the next run downloads them again, and their items land in the dataset twice.

Why not crawler.stop_async() for both events

• A container exiting with code 0 during a migration ends the run as SUCCEEDED and it isn't restarted (only a non-zero exit with wasMigrationInitiated re-schedules it), so stopping the crawl on MIGRATING would lose the pending work.
• On a graceful abort the run is already DONE, so exiting with 0 yields ABORTED.

Migration

• ApifyScheduler listens for MIGRATING: next_request() hands out nothing, and the requests in flight are marked as handled as they finish.
• The crawl stays idle until the platform kills it, so the next run continues with the pending requests only.
• Actor.reboot() awaits the same listener, so a reboot waits for the requests in flight to settle first.

Abort

• New ApifyGracefulStopExtension (registered by apply_apify_settings) calls crawler.stop_async() on ABORTING.
• The scheduler marks the requests in flight as handled as they finish, and the run ends as ABORTED within the grace period.

Docs and tests

• The Scrapy guide describes the migration behavior and the start-URL caveat: the default Spider.start() yields with dont_filter=True, so a restarted run crawls the start URLs again. The example spider overrides start() with plain requests.
• The start page now deduplicates against the same link on subpages, so the title-spider e2e test asserts unique items.
• New e2e tests: a graceful abort with a request in flight, and an Actor.reboot() mid-crawl.

✍️ Drafted by Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.83%. Comparing base (5a0a752) to head (dcbc9a3).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1104      +/-   ##
==========================================
+ Coverage   92.65%   92.83%   +0.18%     
==========================================
  Files          51       53       +2     
  Lines        3445     3519      +74     
==========================================
+ Hits         3192     3267      +75     
+ Misses        253      252       -1     

Flag	Coverage Δ
e2e	32.73% <0.00%> (-0.71%)	⬇️
integration	53.33% <0.00%> (-1.15%)	⬇️
unit	89.42% <100.00%> (+0.25%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.