How to use a proxy with Scrapy

What Scrapy gives you

Scrapy's documentation says HttpProxyMiddleware sets the proxy for requests through the proxy meta value on Request objects. Like Python's urllib.request module, it also obeys the http_proxy, https_proxy and no_proxy environment variables. You can set the proxy meta key per request to a value like http://some_proxy_server:port or http://username:password@some_proxy_server:port. The documentation adds that the meta value takes precedence over the environment variables and ignores no_proxy.

Any proxy provider works this way. You need a host, a port and a login or IP authorization. The placeholders here are PROXY_HOST, PROXY_PORT, PROXY_USER and PROXY_PASS. Only use proxies for lawful purposes and within the terms of the websites you visit. A proxy is not a cloak: the proxy provider and the sites you visit can still see activity, so do not treat a proxy as making your traffic anonymous.

What Webshare's documentation says about connecting

We use Webshare as the example because it is a common starting point and has a free plan. We have not tested it. The details below come from Webshare's help center article on the rotating proxy endpoint and its API documentation on proxy connections, as read on October 31, 2026. Your own dashboard shows your real values, so copy them from there and not from this page.

Rotating endpoint: the help center says the domain name is p.webshare.io, the default port is 80, and the alternative ports are 1080, 3128, or any port between 9999 and 29999. It describes this method as rotating proxies randomly within your list for each request. Direct connections are different: the API docs say they use the proxy address and port fields from the Proxy List API, so each proxy in your list has its own address and port.

Authentication: the API docs describe two ways. With username and password authentication you provide the username and password along with the proxy address and port. With IP authorization only the address and port are needed, with no credentials, because Webshare recognizes the IP address you connect from.

Username options: for backbone connections the API docs say you can add parameters to your username with hyphens, for example a two-letter country code, a numeric session ID for a sticky session, or the word rotate for a new IP on every request. We have not checked which of these your plan includes, so check your dashboard before relying on them.

Option 1: set the proxy on a request

The most direct approach is to put the proxy in the request's meta. This spider requests a page that shows your IP address, so you can confirm the proxy is used. Credentials are written in the proxy URL, as the Scrapy documentation describes.

If you do not have a proxy yet, Webshare has a free plan you can try (Affiliate link) and shows your host, port and login in its dashboard. Check its current plan limits and terms on its own site first, as they can change.

import scrapy


class IpSpider(scrapy.Spider):
    name = "ip_check"

    def start_requests(self):
        yield scrapy.Request(
            "https://api.ipify.org?format=json",
            meta={"proxy": "http://PROXY_USER:PROXY_PASS@PROXY_HOST:PROXY_PORT"},
        )

    def parse(self, response):
        self.logger.info("IP seen by the site: %s", response.text)

Run it from inside a Scrapy project with scrapy crawl ip_check. Do not leave real credentials in a spider file you commit.

Option 2: a small downloader middleware for every request

Setting meta on each request gets repetitive. A downloader middleware can add the proxy to all requests, and it lets you read credentials from environment variables. Set PROXY_USER, PROXY_PASS, PROXY_HOST and PROXY_PORT in your shell before running.

The middleware sets request.meta["proxy"], and the built-in HttpProxyMiddleware then handles the credentials in the URL. Pick an order number lower than the built-in HttpProxyMiddleware, so your middleware sets the meta value before the built-in one reads it. The 350 in the settings example is our suggestion; check the default middleware order list in the Scrapy documentation for your version.

# middlewares.py
import os


class ProxyMiddleware:
    def process_request(self, request, spider):
        user = os.environ["PROXY_USER"]
        password = os.environ["PROXY_PASS"]
        host = os.environ["PROXY_HOST"]
        port = os.environ["PROXY_PORT"]
        request.meta["proxy"] = f"http://{user}:{password}@{host}:{port}"

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 350,
}

# Be gentle with the sites you crawl.
DOWNLOAD_DELAY = 1
CONCURRENT_REQUESTS_PER_DOMAIN = 4
ROBOTSTXT_OBEY = True

Option 3: environment variables

Because HttpProxyMiddleware obeys the standard variables, you can set HTTP_PROXY and HTTPS_PROXY in your shell and run the spider without any proxy code. The credentials go in the URL, as in the examples above. It is quick for testing. Be careful with it: the requests documentation warns that storing credentials in environment variables or version-controlled files is a security risk, and the same caution applies here. Use a shell session you control and clear the variables afterward.

Rotation and sticky sessions

Webshare's help center describes a rotating endpoint at p.webshare.io, default port 80, with rotation within your list for each request. Its API docs describe adding hyphen-separated parameters to the username for a country, a numeric session ID for a sticky session, or rotate for a new IP on every request. We have not checked what your plan includes, so use the dashboard values.

If your provider gives you a list of direct proxies instead, each with its own address and port, you can pick one per request in your middleware, for example by cycling through a list. Keep the list in a file outside your repository.

A note on SOCKS

The Scrapy documentation says that if the download handler supports it, you can use a SOCKS proxy URL such as socks5://username:password@host:port, and that its HttpxDownloadHandler supports SOCKS while the other built-in handlers do not. For an ordinary setup, use an HTTP proxy URL.

Check that it worked

Compare your public IP address with and without the proxy. Open our what is my IP tool in a normal browser tab and note the address, then run the same check through the proxy. The two addresses should differ if traffic really goes through the proxy. You can also run your proxy through our proxy checker, or look up the address you see with the IP lookup tool to see its country and network. In Scrapy, log response.text from a request to an IP-echo page, as the first spider does.

If the address did not change, the proxy is probably not being used. If you get an error instead, see the troubleshooting section.

Troubleshooting

• 407 responses: the login is wrong or missing in the proxy URL.
• Timeouts: the proxy may be slow or down. Lower concurrency, and raise DOWNLOAD_TIMEOUT only if needed.
• The IP did not change: a different middleware may be overriding your meta value, or the request never reached your code.
• Blocked pages: a site may block known proxy addresses. Slow down, check the site's terms, and do not try to bypass access controls.

Next steps

Scrapy is a good fit for larger crawls, and the guide to proxies for web scraping explains how proxy types differ. For smaller scripts, see Python requests. For JavaScript-heavy pages, see Puppeteer.

Once your proxy works, the next step is choosing a plan that fits your use. See Webshare's current plans (Affiliate link) or compare other providers; the setup above stays the same either way.