Rotating AWS proxy endpoints for Scrapy

It has been a long time since I last wrote on this blog. Of course, a lot has happened and a lot has been built in the meantime. I want to revive the blog with a few project writeups, starting with one that has been sitting around for too long: scrapy-proxygen.

scrapy-proxygen is a Scrapy downloader middleware for assigning, rotating and retiring proxy endpoints. It can work with a normal fixed proxy list, but the more interesting part is that it can also generate managed, short-lived proxies on demand using AWS ECS/Fargate.

Why build this? Link to heading

When crawling with Scrapy, a single proxy endpoint can quickly become a bottleneck, even with polite crawl delays and careful request scheduling.

The basic idea of this project is simple:

AWS can give me proxy endpoints by starting tiny Squid proxy containers (via AWS Fargate tasks). Then, a Scrapy spider can create its own temporary proxy pool while it runs.

Instead of renting a long-lived proxy fleet, scrapy-proxygen can start proxy containers at the beginning of a crawl, rotate requests across them, retire unhealthy ones, and clean everything up afterwards.

The original motivation was mainly cost reduction: Managed proxy providers such as Zyte, one of the leading providers for web crawlers, are convenient, but an AWS proxy pool can be substantially cheaper.

The important pricing difference is that Zyte-style proxy services are usually priced by the number of requests, whereas AWS Fargate is priced by deployment time. With scrapy-proxygen, the relevant quantity is therefore not only “how many requests?”, but also “how busy can I keep each proxy while the crawl is running?”.

Assume the crawl should finish within 72 hours. With the default scrapy-proxygen Fargate task size (0.25 vCPU, 512 MiB), I estimate one proxy task at roughly $0.015 per hour. One proxy running for the full 72 hour window therefore costs only about $1.08 in Fargate compute (eu-west region).

Zyte’s packaging and pricing can change over time, so the Zyte column below should be read as an approximate request-priced reference I used when starting this project.

The very conservative 2 requests/minute/proxy assumption almost removes the advantage. But once the proxies are kept moderately busy, the gap opens up quickly.

Importantly, this calculation does not assume that every proxy endpoint remains healthy for the whole crawl. If a proxy fails request checks, the corresponding proxy task can be stopped and a new one can take its place (with a fresh IP). Replacement only adds a small correction if the slot is unavailable while ECS starts the new task.

What it does Link to heading

The project contains three main pieces:

1. Scrapy middleware
   ProxygenMiddleware assigns proxies to requests, keeps track of proxy health, retries failed requests with another proxy, and updates Scrapy stats.

2. Failure detection middleware
   FailureDetectionMiddleware marks responses or exceptions as request failures. The default policy treats non-200/301/302 responses and empty 200 responses as failures, but spiders can provide custom logic with response_is_failure and exception_is_failure.

3. Proxy providers
   Providers abstract where proxies come from. There is a static provider for fixed proxy lists and an AWS ECS/Fargate provider for dynamic proxy generation.

Internally, proxies move through a small state machine:

• unchecked — known, but not successfully used yet;
• good — recently worked;
• dead — recently failed and waiting for replacement or backoff.

Dead proxies can be reanimated after exponential backoff with jitter, or, in the AWS-backed case, stopped and replaced with a new proxy container (i.e. an AWS Fargate task).

The project started as a rewrite inspired by earlier rotating-proxy middleware for Scrapy, but the AWS provider became the main new direction.

An example Link to heading

The repository ships with a small end-to-end demo in examples/httpbin_proxy_demo. It uses AWS-generated proxy tasks and calls https://httpbin.org/ip repeatedly to verify that requests are routed through the configured proxy endpoint.

The example settings are deliberately small. The helper loads generated PROXYGEN_* values from .proxygen-aws.env, enables ProxygenMiddleware and FailureDetectionMiddleware, and converts environment strings into Scrapy settings:

# examples/httpbin_proxy_demo/httpbin_proxy_demo/settings.py
from proxygen.settings import load_proxygen_settings

PROXYGEN_CLOSE_SPIDER = True
PROXYGEN_PAGE_RETRY_TIMES = 3
PROXYGEN_LOGSTATS_INTERVAL = 10
PROXYGEN_AWS_WAIT_TIMEOUT = 180

load_proxygen_settings(globals())

The spider itself is intentionally tiny. It yields repeated requests to https://httpbin.org/ip, records the response status, the origin reported by httpbin, and the proxy URL Scrapy used for the request:

class HttpbinIpSpider(scrapy.Spider):
    name = "httpbin_ip"

    async def start(self):
        count = int(getattr(self, "count", 10))
        url = getattr(self, "url", "https://httpbin.org/ip")

        for request_number in range(1, count + 1):
            yield scrapy.Request(
                url,
                callback=self.parse_ip,
                cb_kwargs={"request_number": request_number},
                dont_filter=True,
            )

    def parse_ip(self, response, request_number: int):
        data = json.loads(response.text)
        proxy = response.request.meta.get("proxy")

        yield {
            "request_number": request_number,
            "status": response.status,
            "origin": data.get("origin"),
            "proxy": mask_proxy(proxy),
        }

Current status Link to heading

Version 0.1.0 is the first cleaned-up release of the project. I introduced provider abstractions, so adding other cloud providers than AWS is possible.

Code and documentation are on GitHub:

github.com/n1kn4x/scrapy-proxygen

As always with scraping infrastructure: use it responsibly, respect robots.txt and terms of service, and do not use proxies to abuse services.