[Blog](https://scrapfly.io/blog) / [frameworks](https://scrapfly.io/blog/tag/frameworks) / [Web Scraping With Scrapy: The Complete Guide in 2026](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) # Web Scraping With Scrapy: The Complete Guide in 2026 by [Bernardas Alisauskas](https://scrapfly.io/blog/author/bernardas) Sep 02, 2026 18 min read [\#frameworks](https://scrapfly.io/blog/tag/frameworks) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [\#scrapy](https://scrapfly.io/blog/tag/scrapy) [\#xpath](https://scrapfly.io/blog/tag/xpath) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy&text=Web%20Scraping%20With%20Scrapy%3A%20The%20Complete%20Guide%20in%202026 "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-scrapy) A Scrapy tutorial is only useful when the setup, spider, pipeline path and the export command all belong to the same project. Copy one snippet from one place and another from somewhere else and you end up with a messy pipeline and an output file that never gets written. This guide keeps everything in one Scrapy 2.18 project with one spider named`quotes` and an output file named `quotes.jsonl`. Every command and every block continue the same project, so you can paste into a clean environment in order and watch the crawl run! ## Key Takeaways You need Python 3.10 or newer and `Scrapy==2.18.0`. The scraping target is [Quotes to Scrape](https://quotes.toscrape.com/), a small site published for practice exactly like this. - Scrapy 2.18 starts custom requests from an asynchronous `start()` generator, then continues through normal callbacks. - Test CSS or XPath selectors in `scrapy shell` before adding them to a spider. - Keep item validation in a pipeline and enable it with the exact project module path. - Diagnose the response before adding a proxy or browser. A 403, a 429, a challenge body, and a JavaScript-empty page each need a different fix. With the promise set, the next section describes the finished artifact so you know what every later step is working toward. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## What will you build with Scrapy 2.18? You will build a paginated quotes spider with three fields, one validation pipeline and a JSON lines export. The spider starts at the first page of quotes to scrape, extracts every quote, follows the next page link until there are none left then writes one JSON object per line to `quotes.jsonl`. Here is the final project tree with only the files used in this guide: ``` tutorial ├── scrapy.cfg └── tutorial ├── __init__.py ├── items.py ├── middlewares.py ├── pipelines.py <- ValidateQuotePipeline lives here ├── settings.py <- pipeline is enabled here └── spiders ├── __init__.py └── quotes.py <- QuotesSpider lives here ``` Use python 3.10 or newer and pin `Scrapy==2.18.0` so the `async def start()` interface and the settings values in this guide match what you run. Quotes to Scrape is the target for a reason. Its markup is stable and built for training, sp the CSS selectors below keep working. Nothing here depends on a guessed selector for a real production site that could change next week. The output schema has three keys: - `text` is the quote body. - `author` is the person credited. - `tags` is a list of tag strings. Simplified relation between Scrapy's Crawler and a project's SpiderThe diagram shows the split you work with. The Crawler runs the engine, the scheduler, and the downloader. Your Spider supplies the requests to send and the callbacks that turn responses into items. The next section walks that flow in order. ## How does a Scrapy crawl process requests and items? A Scrapy crawl is a loop. The spider yields requests into a scheduler, the downloader fetches each one and hands the response to the callback that requested it. Scrapy is built on [Twisted](https://twisted.org/), an event networking engine, so the loop is concurrent without you writing thread code. ### How do Scrapy requests, responses, callbacks, and items connect? A spider yields two kinds of objects: - A `Request` names a URL and a callback. - A plain dictionary or item is a result that goes to the pipelines and then to the feed. Callbacks receive a `Response`, and inside one you work with a few methods: - `response.css(...)` or `response.xpath(...)` pull data out of the page. - `response.follow(link, callback=...)` queues another page. - `response.follow()` resolves a relative href against the current page URL, so you never rebuild the base URL by hand. Two selector methods decide how you read a value: - `.get()` returns the first match as a string, or `None` when there is no match. - `.getall()` returns every match as a list. That is why `tags` uses `.getall()` while the single `text` and `author` fields use `.get()`. ### Where do Scrapy downloader middleware and item pipelines run? Downloader middleware sits between the scheduler and the downloader. It sees every request before it leaves and every response before the callback. That makes it the place for retry rules, proxy selection and header policy. Item pipelines sit after the callbacks. Every item a callback yields passes through each enabled pipeline in priority order before it reaches the feed. This makes pipelines the place for validation and transformation. This project adds a pipeline and leaves middleware alone. When you do need custom middleware, the [dedicated Scrapy middleware guide](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them) owns the implementation detail. Next you create the environment and the project. ## How do you install Scrapy 2.18 and create a project? ### How do you create and verify a Scrapy environment? 1. Create a virtual environment 2. Activate it 3. Install the pinned version then confirm it shell```shell python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install "Scrapy==2.18.0" scrapy version ``` Do not install a `latest` or unbounded `scrapy`, because a different minor version can change default settings and the startup interface used below. ### What files does scrapy startproject tutorial create? Create the project and enter its directory: shell```shell scrapy startproject tutorial cd tutorial ``` That writes a `scrapy.cfg` deployment file and an inner `tutorial` package. Three files in that package matter for this guide: - `tutorial/settings.py` holds project configuration, including which pipelines are enabled. - `tutorial/pipelines.py` is where `ValidateQuotePipeline` goes. - `tutorial/spiders/` is the package your spider module goes in. The other generated files: `items.py` and `middlewares.py` stay untouched here. Next write the spider. ## How do you build a Scrapy 2.18 spider with async start()? ### How does a Scrapy spider request its first page? Create `tutorial/spiders/quotes.py`. In Scrapy 2.18 a spider issues its first requests from an asynchronous `start()` generator rather than the old `start_requests()` method: python```python # tutorial/spiders/quotes.py import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" async def start(self): yield scrapy.Request( "https://quotes.toscrape.com/", callback=self.parse, ) def parse(self, response): ... ``` `name = "quotes"` is how the CLI refers to this spider. `async def start()` yields the first request and points it at `self.parse`. After that first request the crawl continues through normal callbacks, so `parse` does not need to be async. ### How do you parse quote fields with Scrapy CSS selectors? Fill in `parse` in the same class. Each quote on the page is a `div.quote` block and the three fields sit at known selectors inside it: python```python def parse(self, response): for quote in response.css("div.quote"): yield { "text": quote.css("span.text::text").get(), "author": quote.css("small.author::text").get(), "tags": quote.css("div.tags a.tag::text").getall(), } ``` The loop yields one plain dictionary per quote with the three declared keys. `text` and `author` use `.get()` because each quote has exactly one. `tags` uses `.getall()` because a quote carries several tag links and you want the whole list. ### How do you follow pagination with response.follow()? Quotes to Scrape puts the next-page link in `li.next a`. Read that href and follow it only when it exists: python```python next_page = response.css("li.next a::attr(href)").get() if next_page is not None: yield response.follow(next_page, callback=self.parse) ``` `response.follow()` turns the relative `/page/2/` href into an absolute request, so you never concatenate the base URL yourself. The same `parse` callback handles every page, and the `if` check ends the crawl when the last page has no next link. Before running this confirm the selectors against the live page. ## How do you test Scrapy selectors before a crawl? ### How do you use scrapy shell on Quotes to Scrape? `scrapy shell` fetches a page once and drops you into a python prompt with a live `response`. Check every selector the spider uses before you trust it: shell```shell scrapy shell "https://quotes.toscrape.com/" ``` python```python response.status response.css("div.quote") quote = response.css("div.quote")[0] quote.css("span.text::text").get() quote.css("small.author::text").get() quote.css("div.tags a.tag::text").getall() response.css("li.next a::attr(href)").get() ``` Run each line and read the return value. If `response.css("div.quote")` is an empty list then the page structure changed or the fetch was blocked and there is no point running the full crawl yet. ### How should a Scrapy spider handle missing fields? `.get()` returns `None` when a selector matches nothing and `.getall()` returns an empty list. The spider does not crash on a missing field. It just yields an item with a `None` or `[]` value. Deciding what counts as a broken item is a separate job. Keep that rule in one pipeline rather than scattering `if` checks across selectors which is what the next section does. ## How do you validate Scrapy items in a pipeline? ### How does ValidateQuotePipeline reject incomplete items? Add this class to `tutorial/pipelines.py`. It uses `ItemAdapter` so it works whether the spider yields a dict or an item class and it raises `DropItem` from `scrapy.exceptions` to discard anything missing a required field: python```python # tutorial/pipelines.py from itemadapter import ItemAdapter from scrapy.exceptions import DropItem class ValidateQuotePipeline: def process_item(self, item): adapter = ItemAdapter(item) if not adapter.get("text"): raise DropItem("missing text") if not adapter.get("author"): raise DropItem("missing author") return item ``` In Scrapy 2.18 `process_item` takes `self` and `item`. The old `process_item(self, item, spider)` signature still runs but is deprecated, so new code drops the `spider` argument. A dropped item is logged and left out of the feed. `tags` is allowed to be empty because a quote without tags is still a valid record. ### How do you enable a Scrapy pipeline with the correct module path? A pipeline does nothing until it is listed in `ITEM_PIPELINES`. Add this to `tutorial/settings.py`: python```python # tutorial/settings.py ITEM_PIPELINES = { "tutorial.pipelines.ValidateQuotePipeline": 300, } ``` A wrong prefix is the classic reason a pipeline silently never runs. The `300` is priority: lower runs earlier, and any 0 to 1000 value is fine for one pipeline. Now run the crawl. Scrapfly #### Extract structured data automatically? Scrapfly's Extraction API uses AI to turn any webpage into structured data — no selectors needed. [Try Free →](https://scrapfly.io/register)## How do you run, debug, and export a Scrapy crawl? ### How do you export Scrapy items as JSON Lines? Run the spider and send items straight to a file: shell```shell scrapy crawl quotes -O quotes.jsonl python -c "import json; print(json.loads(open('quotes.jsonl', encoding='utf-8').readline()))" ``` Capital `-O` overwrites the file each run, lowercase `-o` appends and quietly doubles your data. The `.jsonl` extension makes Scrapy write one JSON object per line, so nothing has to hold a whole array in memory. The second command parses the first line to confirm the `text`, `author`, and `tags` keys. ### What should you inspect when a Scrapy crawl exports no items? Work through the response before you touch concurrency or reach for a browser: - Check the final URL and the response status in the log. A redirect to a login or a 403 explains an empty file immediately. - Re-run the exact selectors in `scrapy shell`. A structure change shows up here in seconds. - Read the stats line at the end of the run. `item_scraped_count` at zero with `response_received_count` above zero means fetching worked and parsing did not. - Check whether the pipeline dropped everything. `DropItem` lines in the log point at a validation rule that is too strict. - Confirm the file you are opening is the file the crawl wrote. A stale `quotes.jsonl` in another directory is a common false alarm. Once the crawl exports clean JSON Lines the settings section covers what to review before pointing a spider at a harder target. ## Which Scrapy defaults and starter settings should you review before production? Scrapy 2.18 ships runtime fallbacks and `scrapy startproject` writes a `settings.py` with more conservative values for several controls. Inspect the effective settings for your project before you change anything because neither the runtime fallback nor the fresh-project value is a universal production recipe. ### How do project settings, custom\_settings, and CLI overrides interact? Three layers stack, from lowest priority to highest: 1. Project `settings.py` is the baseline. 2. A spider's `custom_settings` dict overrides the project for that spider only. 3. A CLI `-s NAME=value` flag overrides both for that one run. To see what actually applies, ask Scrapy: shell```shell scrapy settings --get CONCURRENT_REQUESTS_PER_DOMAIN ``` For the full list of settings and their meanings, use the [official Scrapy 2.18 settings reference](https://docs.scrapy.org/en/2.18/topics/settings.html) ### Which eight Scrapy controls deserve an explicit production decision? | Control | Runtime fallback / fresh project | Review when | Main tradeoff | |---|---|---|---| | `CONCURRENT_REQUESTS_PER_DOMAIN` | `8` / `1` | Throughput, latency, or 429/403 rates move | More throughput also increases simultaneous target load | | AutoThrottle and delay bounds | disabled, `DOWNLOAD_DELAY=0` / disabled, delay `1` | Static pacing does not fit changing latency | Adaptive timing still needs a hard cap and measured bounds | | `RETRY_TIMES` and `RETRY_HTTP_CODES` | `2`; 500, 502, 503, 504, 522, 524, 408, 429 | Failures are demonstrably transient | Extra attempts multiply latency and target load; 403 is not included | | `DOWNLOAD_TIMEOUT` | `180` seconds / inherited | Dead requests occupy slots too long | Short limits reject valid slow responses | | `JOBDIR` | `None` / inherited | A long crawl must pause and resume cleanly | Disk state adds I/O, serialization, and same-version constraints | | `DEPTH_PRIORITY` plus queue classes | `0` with normal LIFO queues / inherited | Breadth coverage matters more than finishing one branch | Wider frontiers increase memory and disk pressure | | `ROBOTSTXT_OBEY` | `False` / `True` | The crawl's explicit policy is being set | Changing it alters scope, not performance alone | | `RFPDupeFilter` and request fingerprints | URL, method, and body; headers and fragments ignored / inherited | Headers, tenants, or known noise parameters change resource identity | Broader fingerprints increase revisits; narrower ones drop distinct work | For each row, decide from four things: - The failure symptom that makes you look. - The rule that tells you which way to move. - One concrete configuration line. - The cost you accept by changing it. These boundaries hold no matter what the traffic looks like: - `RETRY_TIMES=2` means two retries after the first download attempt. Do not add 403 to `RETRY_HTTP_CODES` without target-specific evidence that the 403 is transient. - AutoThrottle's target concurrency is an average, not a hard ceiling. The per-domain cap and the delay bounds still apply on top of it. - Breadth-first order needs a positive `DEPTH_PRIORITY` plus FIFO disk and memory queues. Concurrency makes the resulting order approximate, not exact. - `JOBDIR` persists the scheduler queue, the duplicate filter, and simple spider state across a clean stop. It is not crash-proof checkpointing and not durable cookie storage. - Keep `ROBOTSTXT_OBEY=True` unless the crawl owner has an explicit, target-specific policy. The generated value is a deliberate default, not a bug. - The default request fingerprint ignores headers and fragments. Set a custom `REQUEST_FINGERPRINTER_CLASS` only for a stable field that genuinely changes resource identity, and do not use the removed `REQUEST_FINGERPRINTER_IMPLEMENTATION` switch. Do not paste a universal settings block from anywhere. The example numbers are starting points to check against your own latency, failure stats, target policy, and completeness goal. When settings are not enough, the next section covers middleware, proxies, and browsers. ## When should you add Scrapy middleware, proxies, or a browser? ### When does a Scrapy downloader middleware belong in the project? Reach for a downloader middleware when a rule applies to every request or response regardless of callback such as retry policy, proxy assignment, header rewriting, or request filtering. Scrapy already runs a stack of built-in middleware, so your class needs a priority number that places it correctly against the retry and redirect middleware. ### When should a Scrapy spider use a proxy route? Use a proxy when the target blocks or rate-limits your IP, or when you need requests to originate from a specific region. Scrapy supports per-request proxies through request meta and project-wide proxies through middleware, plus authenticated gateways, sticky sessions, and proxy-aware retry rules. [The Scrapy proxy rotation guide](https://scrapfly.io/blog/answers/scrapy-spiders-proxy-rotation) covers all of those patterns. ### When does a Scrapy response require a browser? A browser is only relevant when the data you need is not in the HTML that Scrapy downloads because it appears after JavaScript runs or after a user interaction. Check with `scrapy shell` first. If `response.text` already contains the data or a JSON blob you can parse then you do not need a browser. - [Scrapy with Playwright](https://scrapfly.io/blog/posts/how-to-use-scrapy-with-playwright) is the current recommended browser route. - [Scrapy with Selenium](https://scrapfly.io/blog/posts/web-scraping-dynamic-web-pages-with-scrapy-selenium) covers Selenium maintenance work. - [Scrapy with Splash](https://scrapfly.io/blog/posts/web-scraping-with-scrapy-splash) is a legacy compatibility guide. Rendering JavaScript does not bypass anti-bot protection. A rendered page can still be served a challenge, which is the subject of the next section. ## Why is a Scrapy spider getting blocked? ### How do you classify a Scrapy 403, 429, challenge body, or empty page? Different blocks look different in the response. Read the status, the headers and the body before you decide on a fix: | Observed symptom | Inspect | Likely class | Next route | |---|---|---|---| | HTTP 403 from the target | `response.status`, body snippet, request headers sent | IP or header reputation, or anti-bot fingerprinting | Rotate IP and fix headers, then a managed anti-bot route | | HTTP 429 from the target | `response.status`, `Retry-After` header, current request rate | Rate limiting | Lower `CONCURRENT_REQUESTS_PER_DOMAIN`, raise the delay, enable AutoThrottle | | Status 200 with CAPTCHA or challenge HTML | body content, page title | Challenge interstitial | CAPTCHA handling, then a managed anti-bot route | | Status 200 but target data missing from HTML | `scrapy shell`, `response.text` length, `scrapy view` | Client-side rendering | Browser route, or find the JSON API the page calls | | HTTP 407 on the request | proxy username, password, gateway URL | Proxy authentication failure | Fix proxy credentials or gateway config | | Timeout or 5xx response | `DOWNLOAD_TIMEOUT`, retry stats, target status page | Transient target or network fault | Bounded retries, then back off | Route by the symptom you actually see. A 403 and a challenge body served with status 200 are different problems even though both mean blocked. For the details behind each row, see [403 diagnosis](https://scrapfly.io/blog/posts/403-forbidden-web-scraping), [CAPTCHA handling](https://scrapfly.io/blog/posts/how-to-bypass-captcha-while-web-scraping), and [anti-blocking mechanisms](https://scrapfly.io/blog/posts/how-to-scrape-without-getting-blocked-tutorial). ### When should Scrapy work route to a standalone Scrapfly product When the blocking is persistent and you would rather not maintain proxy pools and fingerprint patches yourself then move the fetching layer out of scrapy. - Route HTTP fetching, JavaScript rendering, proxy selection, and anti-bot handling to the [Web Scraping API](https://scrapfly.io/products/web-scraping-api). - Route click, form, and long-lived browser workflows to [Cloud Browser](https://scrapfly.io/products/cloud-browser-api). Both are standalone HTTP services. Your Scrapy spider keeps its selectors and pipeline and calls the API for the response. There is no Scrapy SDK to install and no supported in-framework integration to configure. The next section covers scaling the crawl itself. ## How do you run Scrapy beyond one local spider? Once one spider runs cleanly four directions come up. Each has a primary source worth reading before you commit to an architecture. For distributed crawls, [scrapy-redis](https://github.com/rmax/scrapy-redis) shares the request queue and the duplicate filter across worker processes through Redis, so several machines crawl one frontier. Plan for queue persistence and a way to reset state between runs. For scheduling, an [Apache Airflow](https://airflow.apache.org/) DAG can trigger `scrapy crawl` on a cron-like schedule and track success and failure per run. Keep the crawl itself a single command so the orchestration layer stays thin. For deployment, a Docker image that pins Python and `Scrapy==2.18.0` and runs the crawl as its entrypoint gives you a reproducible unit to run anywhere. Mount or ship the output rather than leaving it in the container. For tests, Scrapy [spider contracts](https://docs.scrapy.org/en/2.18/topics/contracts.html) check a callback against a live URL, and [Spidermon](https://spidermon.readthedocs.io/) validates item counts and field coverage at the end of a run. Both catch a silently broken selector before it reaches your data. None of these needs custom framework code to get started. The FAQ covers two questions that come up while building the spider itself. ## FAQ Should you use Scrapy or Beautiful Soup?Scrapy is a crawling framework with a scheduler, a downloader, retries, and pipelines. Beautiful Soup is only an HTML parser, so you pair it with a request library and write the crawl loop yourself. Use Scrapy when you need to follow links at scale and see [Scrapy vs Beautiful Soup](https://scrapfly.io/blog/answers/scrapy-vs-beautifulsoup) for the full comparison. Can Scrapy callbacks use async code?Yes. A `parse` callback can be `async def,` and inside it, you can `await` a coroutine or iterate an async generator, which is useful for calling an async API or database while parsing. The [Scrapy 2.18 coroutine documentation](https://docs.scrapy.org/en/2.18/topics/coroutines.html) lists which callback and middleware results may be awaited. Can you use Selenium with Scrapy?Yes, but the two have different architectures, so the integration adds a browser process per request and needs a middleware to bridge them. Only reach for it when the data truly requires a rendered page. See [Scrapy with Selenium](https://scrapfly.io/blog/posts/web-scraping-dynamic-web-pages-with-scrapy-selenium) for the setup, or [Scrapy with Playwright](https://scrapfly.io/blog/posts/how-to-use-scrapy-with-playwright) for the current recommended route. How do you scrape dynamic pages with Scrapy?Check where the data actually lives first. A lot of dynamic content ships as JSON inside a `