Choosing proxies for web scraping without getting blocked

Start with the rules, not the proxy

Web scraping means collecting data from web pages with a program instead of a person. It is common and often legitimate: price monitoring, academic research, search and archive projects, and checking your own pages. It also carries obligations. Before choosing a proxy, check whether the site offers an official API or a data download. An API is more stable, easier to use and clearly permitted, and it removes the need to work around anything.

If there is no API, read the site's terms of service and its robots.txt file. These two documents tell you what the owner expects, and they are the first things anyone will point to if there is a dispute.

robots.txt and terms of service

The robots.txt file, found at the root of a site such as example.com/robots.txt, tells automated programs which paths they are asked not to fetch and sometimes how fast they may go. The format is documented in RFC 9309. It is a convention, not a lock. Nothing technically stops you from ignoring it, but doing so is widely regarded as bad practice and can be used as evidence of bad faith. Respect it, including any crawl-delay hint.

The terms of service are different. They are a contract that you may have agreed to by using the site or creating an account, and many forbid automated collection. Breaking them is generally a civil matter rather than a criminal one, but the consequences can include a banned account, a cease-and-desist letter or a lawsuit. If you must log in to see the data, terms are much more likely to apply to you.

Legal caveats in general terms

This section is general information, not legal advice, and the law differs by country and by facts. If a project matters commercially, talk to a qualified lawyer.

• Computer misuse laws. In the United States, the Computer Fraud and Abuse Act (CFAA) prohibits accessing a computer without authorization or beyond authorized access. Court decisions in recent years have narrowed how it applies to public web pages, but bypassing a login, a technical barrier or an explicit block remains the riskiest area. Other countries have their own equivalents.
• Personal data. If you collect information about identifiable people, such as names, profiles or contact details, data protection law can apply even when the data is public. In the European Union and the United Kingdom, the GDPR requires a lawful basis, limits on how long you keep data, and respect for people's rights. Collecting less data is the safest approach.
• Copyright and database rights. Facts are generally not protected, but the way a site presents them, its text, images and in some regions its database as a whole, can be. Republishing scraped content is a different risk from analysing it.
• Contract and trespass claims. Sites sometimes rely on terms of service or on claims that heavy automated traffic harms their servers.

How sites decide to block you

Sites defend themselves with a mix of signals. The most common is request volume: too many requests from one address in a short time. Others include an unusual pattern, such as fetching pages at perfectly regular intervals, never loading images or scripts, or requesting pages in an order no person would. They also check the address owner, since traffic from hosting companies is more suspicious on some sites, and whether your headers look like a real browser. You can see what a standard browser sends on our HTTP headers page.

Notice that most of these are about how you behave. A fresh proxy address will not help a scraper that sends hundreds of requests a second, and spreading the same behaviour across many addresses does not make the load on the site any smaller.

Rate limits and polite crawling

The most effective way to avoid blocks is to be a considerate visitor. Keep concurrency low, add a pause between requests, and slow down further when the site slows down. Treat these signals as instructions:

• HTTP 429 (Too Many Requests) means stop and slow down. If the response includes a Retry-After header, wait at least that long.
• HTTP 503 or growing response times can mean the server is struggling. Back off, using a delay that grows after each failure.
• A sudden wall of CAPTCHAs or blocks means you have been noticed. Do not escalate. Reduce your rate or stop.

Also cache what you have fetched so you never ask for the same page twice, run large jobs at quiet times for the site, and send a user-agent string that identifies your project with a way to reach you when that is appropriate. Sites are far more tolerant of crawlers that identify themselves and behave well.

What proxy rotation does

Rotation means sending requests through different proxy addresses instead of one. A rotating setup can change the address on every request or after a set time. A sticky session keeps the same address for a period, which matters when a site expects one visitor to stay on one address while moving through several pages, for example a multi-step search.

Rotation spreads requests across addresses so that no single one trips a per-address limit. That is useful for large, legitimate collection jobs. It is not a licence to exceed what a site allows, and it does not make evasion of a clear block acceptable. If a site has explicitly told you no, rotating addresses to get around it is where legal and ethical risk climbs most.

Which proxy type fits scraping

• Datacenter proxies are fast and cheap and work on sites that do not filter hosting ranges. Start here for public pages with permissive rules. See datacenter proxies.
• Residential proxies are harder for sites to block by network type and let you see location-specific content, but are slower, usually billed per gigabyte, and raise sourcing questions. See residential proxies.
• Mobile proxies suit mobile-only content and are the most expensive. See mobile proxies.

If the data you need is on a site that blocks everything except real consumer networks, ask whether that site is signalling that it does not want automated access. Our guide to residential vs. datacenter proxies goes into the trade-offs.

Where to get proxies

If you need datacenter proxies for a permitted job, Webshare (webshare.io (Affiliate link)) is one provider that offers a free starting tier of 10 proxies, according to its website. Try them in the checker first and read its current terms. GlobeProxy earns a commission if you buy through that link, which does not change what this guide says; see the affiliate disclosure. For pricing, limits and gotchas, read our research-based Webshare review.

Test your proxies before a big run

A job that fails halfway through wastes time and can hammer a site with retries. Before running at scale, test a sample of your proxies with the proxy checker. Confirm each one connects, exits in the country you paid for, belongs to the type of network you expected, and does not pass your real address in headers. Then check the exit address in the IP lookup. Drop any that fail, and test again from time to time, because proxy lists change.

A responsible scraping checklist

• Is there an official API or dataset? Use it.
• Have you read the terms of service and robots.txt, and does your plan comply?
• Do you need personal data? If not, do not collect it. If so, know your lawful basis.
• Is your request rate low enough that the site would not notice a difference in load?
• Do you handle 429 and 503 responses by slowing down?
• Are you avoiding logins, paywalls and CAPTCHAs that were put there to keep automated tools out?
• Do you store only what you need, secure it, and delete it when finished?
• Have you asked a lawyer if the project is commercial or sensitive?