Proxies for Scrapy
The framework handles most of this for you, which is exactly why the two settings that matter get left at their defaults.
The address belongs in middleware
Setting it per request scatters the logic across the spider. In middleware it applies once and stays consistent.
That is also the place to rotate: pick the next address in order, skipping any currently in a failed state.
Concurrency by measurement
Raise concurrent requests in steps and watch the completion time. The curve flattens at some point and only the failure rate keeps climbing.
The number just below that flattening is your setting.
Turn on resume before the first big run
The built-in job directory lets a run continue from where it stopped. Enable it before you need it, because afterwards is too late.
Retries
Three attempts with a growing pause. The framework default is close to right; the pause is the part worth adjusting.
CONCURRENT_REQUESTS = 32 RETRY_TIMES = 3 JOBDIR = "crawls/run-01"