Crawl Control — media crawler dashboard

A local web dashboard to start, stop, and watch a media-downloading crawl in real time.

Setup

pip install -r requirements.txt

Run

python3 server.py

Then open http://127.0.0.1:5000 in your browser.

What you get

• Control panel — set seed URLs, crawl depth, page/file limits, whether to allow other domains, delay between requests, and robots.txt behavior. Click Start crawl.
• Domain radar — pings a new dot each time the crawl discovers a domain it hasn't seen before.
• Gauges — live pages crawled, files downloaded, data pulled, domains seen, elapsed time.
• Activity log — a scrolling console of what the crawler is doing right now.
• Downloaded media tab — a grid of everything downloaded so far, with thumbnails for images, filterable by media kind and source domain. Click a card's domain link to open the page it came from, or "open file" to view the saved file directly.
• Stop crawl — halts the crawl in progress (it finishes its current request first, so there can be a brief delay of up to your configured "delay" setting).

Files are saved under ./runs/<timestamp>/media/, alongside a manifest.json and manifest.csv mapping each file to the page it was found on — same as the CLI version.

Notes

• Only one crawl runs at a time; starting a new one while one is active is blocked until you stop it.
• The dashboard polls the server once a second — no need to keep the tab in any special state, just leave it open.
• Same responsible-use notes as before: this respects robots.txt by default, and downloaded media may be copyrighted — downloading isn't the same as having the right to reuse or redistribute it.