Proxies for Python requests
The library is forgiving enough that mistakes here stay invisible until the dataset turns out to be half empty.
One session per run
Create the session once and reuse it. A new session per request spends time re-establishing the connection every time.
Set the address on the session so every call inherits it, and you cannot forget one.
Retries with a growing pause
Two seconds, then four, then eight. Three attempts is enough; past that the problem is not the connection.
Retry on connection errors and on the codes that mean "later", never on the ones that mean "no".
A success code is not data
A page can return two hundred and contain nothing you need. Check the response size and the presence of a required markup fragment.
Without that check, empty pages accumulate silently and the run reports success.
Write in batches
Collect a thousand rows and write them at once. Row-by-row writing becomes the bottleneck long before the network does.
s = requests.Session()
s.proxies = {"http": PROXY, "https": PROXY}
r = s.get(url, timeout=20)
if len(r.content) < 500: retry()