[Blog](https://scrapfly.io/blog) / [debugging](https://scrapfly.io/blog/tag/debugging) / [Scrapy Error Messages: What They Mean and How to Fix Them](https://scrapfly.io/blog/posts/scrapy-error-messages) # Scrapy Error Messages: What They Mean and How to Fix Them by [Mohab Yousry](https://scrapfly.io/blog/author/mohab-yousry-9396552a) Sep 11, 2026 17 min read [\#debugging](https://scrapfly.io/blog/tag/debugging) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapy](https://scrapfly.io/blog/tag/scrapy) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fscrapy-error-messages "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fscrapy-error-messages&text=Scrapy%20Error%20Messages%3A%20What%20They%20Mean%20and%20How%20to%20Fix%20Them "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fscrapy-error-messages "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fscrapy-error-messages) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fscrapy-error-messages) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fscrapy-error-messages) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fscrapy-error-messages) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fscrapy-error-messages) A Scrapy traceback usually contains one useful line and thirty lines you mostly will not need. Since Scrapy 2.15, built-in download handlers report a failed DNS lookup as `CannotResolveHostError`, not the older Twisted `DNSLookupError` that many answers still quote. This article is a lookup catalog. Every entry pairs the exact message with the layer that emitted it, one diagnostic check, the smallest fix, and a version boundary. Fragments in angle brackets such as `` or `` change between runs and are never literal output. ## Key Takeaways - Read the last exception line first, then jump to the first frame from your own project. The middle of a Scrapy traceback is framework plumbing. - Scrapy 2.15 and later wrap common network failures in framework-specific exceptions such as `CannotResolveHostError` and `DownloadConnectionRefusedError`. - `Filtered offsite request` and `Dropped: ` can be expected control flow, not a broken crawl. - A stopped Twisted reactor cannot be started again. Schedule crawls on one reactor or run the next crawl in a fresh process. - HTTP 403 and 429 are server responses, not transport failures. Scrapy sends unhandled non-2xx responses to the request errback as `HttpError`; a 200 challenge body still reaches `parse`. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## How Do You Read a Scrapy Traceback? Read the final exception line, identify the logger or component that emitted it then move upward to the first frame inside your project. Work through three steps in order: 1. Copy the exception class and the stable text before any dynamic URL, path, class name, or OS error number. 2. Note where the crawl stopped: before startup, before download, inside middleware, or inside an item pipeline. 3. Capture the environment with `scrapy version -v`, `scrapy list`, and the one setting the message points at. Log level is a weak signal. `Filtered offsite request` is DEBUG and a deliberate `DropItem` is a warning, so use the message text as the real identifier. The [Scrapy debugging guide](https://docs.scrapy.org/en/latest/topics/debug.html) covers `scrapy parse` and the shell for the inspection that comes after this triage. Use the table to route a message to the right section, then read the full entry. | Exact fragment | Layer | First check | |---|---|---| | `Request url must be str` | Request build | `type(url)` | | `Missing scheme in request url` | Request build | `repr(url)` | | `callback must be a callable` | Request build | `callable(callback)` | | `Spider not found` | SpiderLoader | `scrapy list` | | `doesn't define any object named` | Component load | dotted import path | | `installed reactor ... does not match` | Reactor setup | import order | | `ReactorNotRestartable` | Process lifecycle | repeated starts | | `Unsupported URL scheme` | Download handler | parsed scheme | | `process_response must return` | Downloader middleware | every return branch | | `requires a spider argument` | Component API | method signature | | `Filtered offsite request` | OffsiteMiddleware | hostname vs `allowed_domains` | | `Dropped:` | Item pipeline | `DropItem` condition | | `CannotResolveHostError` | DNS / download | hostname resolution | | `DownloadConnectionRefusedError` | TCP / download | host and port | | `cannot import name 'canonicalize_url'` | Legacy scrapy-splash | package matrix | Before you change anything, print what the run actually used: bash```bash scrapy version -v scrapy list scrapy settings --get TWISTED_REACTOR scrapy settings --get CONCURRENT_REQUESTS_PER_DOMAIN ``` Run these commands in the failing environment before comparing messages. Since Scrapy 2.13.3, a generated project sets `CONCURRENT_REQUESTS_PER_DOMAIN` to `1`, while the framework default remains `8`; read the effective value instead of assuming it. If you have no runnable spider to test against, start with: [Web Scraping With Scrapy: The Complete Guide in 2026Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) The rest of this article is one section per layer, starting with errors that occur before a byte leaves your machine. ## Which Scrapy Request Errors Happen Before a Download? Scrapy performs request validation when building a `Request` object before making a DNS lookup or connecting to the target server. Resolve these errors by correcting the callback or request arguments rather than changing downloader settings. Every entry uses the same six labels: literal message, meaning, likely cause, diagnostic check, smallest fix, version boundary. The [request and response reference](https://docs.scrapy.org/en/latest/topics/request-response.html) documents the arguments these messages validate. ### Scrapy Request: `TypeError: Request url must be str, got ` - **Literal message:** `` is dynamic and was `int` in the reproduction: `Request url must be str, got int`. - **Meaning:** Scrapy rejected the URL value while building `Request`. Nothing was scheduled. - **Likely cause:** A numeric ID, `None`, a parsed URL object or a list reached the URL field instead of a string. - **Diagnostic check:** Print `repr(url)` and `type(url)` on the line before the request is created. - **Smallest fix:** Correct the upstream value. Call `str(...)` only for a genuine complete URL, and never turn `None` into `"None"`. - **Version boundary:** Longstanding Request validation; the type name varies. ### Scrapy Request: `ValueError: Missing scheme in request url: ` - **Literal message:** `Missing scheme in request url:` is stable, the value is dynamic for example `Missing scheme in request url: books.toscrape.com/catalogue/page-2.html`. - **Meaning:** The URL is a string but not absolute and has no scheme such as `http` or `https`. - **Likely cause:** A bare host or relative link handed straight to `Request`, or a value broken by string concatenation. - **Diagnostic check:** Print `repr(url)` and inspect `urllib.parse.urlsplit(url).scheme`. - **Smallest fix:** Add the scheme for a real absolute URL, or use `response.follow` and `response.urljoin` for links from a response. - **Version boundary:** Longstanding Request validation; the URL value varies. ### Scrapy Callback: `TypeError: callback must be a callable, got ` - **Literal message:** The reproduced type was `str`: `callback must be a callable, got str`. - **Meaning:** Scrapy cannot call the value you assigned as the request callback. - **Likely cause:** Passing `"parse_detail"` as a string or calling `self.parse_detail()` too early so its return value is passed. - **Diagnostic check:** Evaluate `callable(callback)` and check for stray parentheses on a method reference. - **Smallest fix:** Pass `self.parse_detail` not a string and not the result of calling it. - **Version boundary:** The callable check was added in Scrapy 1.5; the type name varies. Python caught all three errors before the scheduler sees the request, so proxies and retries have no effect. The next group stops the run because Scrapy cannot assemble the crawl. ## Which Scrapy Component and Twisted Reactor Errors Stop Startup? Startup errors mean Scrapy could not find a spider or component or could not reconcile which Twisted reactor the process runs. The target has not been contacted. The [common practices page](https://docs.scrapy.org/en/latest/topics/practices.html) covers the one process and `CrawlerRunner` patterns below. ### Scrapy SpiderLoader: `KeyError: 'Spider not found: '` - **Literal message:** `` is the name you asked for, like `KeyError: 'Spider not found: nonexistent'`. - **Meaning:** `SpiderLoader` did not load a spider whose `name` attribute matches. - **Likely cause:** A typo, the wrong directory or virtual environment, a broken import in the spider module, or an incorrect `SPIDER_MODULES`. - **Diagnostic check:** Run `scrapy list` in the same directory and environment. If the spider is missing, import its module directly to show the hidden import error. - **Smallest fix:** Use a name that `scrapy list` prints, or repair the module path or import. - **Version boundary:** Longstanding `SpiderLoader` error; the requested spider name varies. ### Scrapy Component: `NameError: Module '' doesn't define any object named ''` - **Literal message:** Both dotted names are dynamic, for example `Module 'scrapy.downloadermiddlewares.retry' doesn't define any object named 'RetryMiddlewareX'`. - **Meaning:** Scrapy imported the module named in a setting but could not find the class or function inside it. - **Likely cause:** A typo, a renamed class, a package upgrade that moved the object, or docs for a different version. - **Diagnostic check:** Import the module in the active environment and inspect `getattr(module, name, None)`. - **Smallest fix:** Correct the dotted path, or pin the version you already tested against if a package relocated the component. - **Version boundary:** Longstanding `load_object` error; the module and object names vary. ### Scrapy Reactor: `RuntimeError: The installed reactor () does not match the requested one ()` - **Literal message:** Both reactor paths are dynamic. On Windows: `The installed reactor (twisted.internet.selectreactor.SelectReactor) does not match the requested one (twisted.internet.asyncioreactor.AsyncioSelectorReactor)`. - **Meaning:** Something installed a Twisted reactor before Scrapy could apply the `TWISTED_REACTOR` choice. - **Likely cause:** Importing `twisted.internet.reactor` at module load, often indirectly, or conflicting reactor settings in one process. - **Diagnostic check:** Print `type(reactor)` after your imports and look for an import that pulls in `twisted.internet.reactor` before the crawler is created. - **Smallest fix:** Configure the required reactor before anything imports it, and keep one choice per process. Scrapy has defaulted to `AsyncioSelectorReactor` since 2.13. - **Version boundary:** The installed and requested reactor paths are dynamic and platform dependent. ### Twisted Reactor: `twisted.internet.error.ReactorNotRestartable` - **Literal message:** No message body. The class name is the entire literal. - **Meaning:** Code tried to start a Twisted reactor after it had already run and stopped. - **Likely cause:** A second `CrawlerProcess.start()` or `reactor.run()`, a notebook rerun, or process-start code that runs again on import. - **Diagnostic check:** Search the whole process and its import path for crawl start calls, not only the last frame. The visible script may hold one `process.start()` while an import runs another. - **Smallest fix:** Schedule every crawl before one process start, use `CrawlerRunner` on the existing loop, or run the next crawl in a fresh process. Scrapy 2.15 added an [experimental reactorless asyncio mode](https://docs.scrapy.org/en/latest/topics/asyncio.html) via `TWISTED_REACTOR_ENABLED = False`. - **Version boundary:** This is a Twisted reactor lifecycle rule, not a Scrapy-version boundary. Every message here fires before the engine opens a connection, so reinstalling Scrapy or widening timeouts changes nothing. Once the crawl starts failures move to the space between the scheduler and your callback. ## Which Scrapy Middleware and Download Handler Errors Stop Requests? These failures happen between startup and a usable response reaching your callback. Check the URL scheme, component contract, or network boundary. You will find Scrapy network exceptions listed in the [download handlers reference](https://docs.scrapy.org/en/latest/topics/download-handlers.html). ### Scrapy NotSupported: `Unsupported URL scheme '': no handler available for that scheme` - **Literal message:** is `scrapy.exceptions.NotSupported: Unsupported URL scheme 'gopher': no handler available for that scheme`. - **Meaning:** `DOWNLOAD_HANDLERS` has no active handler for the request's scheme. - **Likely cause:** A malformed URL, a disabled handler, or a custom scheme without a component behind it. - **Diagnostic check:** Check `urlsplit(request.url).scheme` and the effective `DOWNLOAD_HANDLERS` mapping. - **Smallest fix:** Correct the scheme, or configure a real handler for the scheme you meant. - **Version boundary:** Longstanding download-handler lookup error; the scheme and handler reason vary. ### Scrapy Middleware: `scrapy.exceptions._InvalidOutput: Middleware .process_response must return Response or Request, got ` - **Literal message:** `_InvalidOutput`, method name, and the return-type text are stable. The middleware class is dynamic. - **Meaning:** A downloader middleware broke the `process_response` contract. - **Likely cause:** A pass-through branch falls off the end of the method or returns `None` explicitly. - **Diagnostic check:** Add one temporary log line before each `return` in the named `process_response` and run one request. - **Smallest fix:** Return the original `response` when no replacement `Request` or `Response` is needed. - **Version boundary:** `_InvalidOutput` is internal, so its class name may change even though the middleware return contract remains public. ### ScrapyDeprecationWarning: `.() requires a spider argument` - **Literal message:** `ScrapyDeprecationWarning: .() requires a spider argument, this is deprecated and the argument will not be passed in future Scrapy versions. If you need to access the spider instance you can save the crawler instance passed to from_crawler() and use its spider attribute.` - **Meaning:** The component still runs, but its method signature crosses a declared future compatibility boundary. - **Likely cause:** Custom or third-party middleware or pipeline code still accepts the old `spider` positional parameter. - **Diagnostic check:** Check the signature of the named method and its package version. - **Smallest fix:** Drop the parameter when unused. When needed, keep the crawler from `from_crawler()` and read `crawler.spider`. - **Version boundary:** This deprecation warning was introduced in Scrapy 2.14; it is not an exception. ### Scrapy DNS: `scrapy.exceptions.CannotResolveHostError: DNS lookup failed: ` - **Literal message:** The class is stable, the detail is dynamic, for example `DNS lookup failed: no results for hostname lookup: no-such-host.web-scraping.dev.` - **Meaning:** The active download handler could not resolve the hostname. - **Likely cause:** A typo, malformed hostname, an unreachable resolver or a network-specific resolution failure. - **Diagnostic check:** Print the parsed hostname, run a system DNS lookup then compare against a well-known host from the same environment. - **Smallest fix:** Correct the hostname or DNS environment. Raising `DOWNLOAD_TIMEOUT` does not make a nonexistent name resolve. - **Version boundary:** Scrapy-specific download exceptions were added in 2.15.0. Older logs show `twisted.internet.error.DNSLookupError`. ### Scrapy TCP: `scrapy.exceptions.DownloadConnectionRefusedError: Connection was refused by other side: ` - **Literal message:** The class is stable, the OS number and text are dynamic. Windows: `Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it.` Linux typically shows `111: Connection refused`. - **Meaning:** Name resolution succeeded, but nothing accepted the connection at that host and port, or the proxy endpoint refused it. - **Likely cause:** A wrong port, a local service that is not running, an incorrect proxy endpoint, or a closed destination listener. - **Diagnostic check:** Confirm the resolved host and port, then test that endpoint outside Scrapy from the same machine. - **Smallest fix:** Correct or start the endpoint. Retries do not repair a permanently closed port. - **Version boundary:** Added in Scrapy 2.15.0. OS wording varies by platform. A single errback can classify the last two without a separate handler per spider. python```python import scrapy from scrapy.crawler import CrawlerProcess from scrapy.exceptions import CannotResolveHostError, DownloadConnectionRefusedError class DownloadErrorSpider(scrapy.Spider): name = "download_errors" custom_settings = {"RETRY_ENABLED": False} targets = [ "https://web-scraping.dev/product/1", "http://no-such-host.web-scraping.dev/", "http://127.0.0.1:9/", ] async def start(self): for url in self.targets: yield scrapy.Request(url, callback=self.parse, errback=self.on_error, dont_filter=True) def parse(self, response): self.logger.info("ok %s %s", response.status, response.url) def on_error(self, failure): self.logger.warning( "failed url=%s type=%s message=%s", failure.request.url, failure.type.__name__, failure.getErrorMessage(), ) if failure.check(CannotResolveHostError): self.logger.warning(" -> DNS problem, check the hostname and resolver") elif failure.check(DownloadConnectionRefusedError): self.logger.warning(" -> nothing listening, check host, port, and proxy endpoint") if __name__ == "__main__": process = CrawlerProcess() process.crawl(DownloadErrorSpider) process.start() ``` Scrapfly #### Scale your web scraping effortlessly Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data. [Try Free →](https://scrapfly.io/register)After running this spider, the first request returns a normal response. The other two hit the errback, which logs each exception type. This errback receives transport failures and, by default, non-2xx responses wrapped as `HttpError`. To inspect 403 or 429 bodies in `parse`, add `meta={"handle_httpstatus_list": [403, 429]}` to the request. The next section covers two log messages that look like failures but usually are the crawl working as configured. ## Which Scrapy OffsiteMiddleware and DropItem Messages Are Not Failures? Both messages can describe intentional filtering. Fix them only when Scrapy dropped a request or item you expected to keep. ### Scrapy OffsiteMiddleware: `Filtered offsite request to '': >` - **Literal message:** DEBUG text is stable. Domain, method, and URL are dynamic, for example `Filtered offsite request to 'books.toscrape.com': `. - **Meaning:** The default `OffsiteMiddleware` rejected a request whose host is outside `allowed_domains`. - **Likely cause:** An external tracking or ad link, an incomplete `allowed_domains` list, or a cross-domain request you actually want. - **Diagnostic check:** Compare the parsed hostname with `allowed_domains` and decide whether the request belongs in the crawl. - **Smallest fix:** Fix link extraction or the domain list. For one intentional request, set the `allow_offsite` request meta key rather than disabling the middleware. - **Version boundary:** The downloader-middleware form was added in Scrapy 2.11.2, `allow_offsite` in 2.13, and mid-crawl `allowed_domains` updates in 2.18. ### Scrapy DropItem: `Dropped: ` - **Literal message:** The log is `Dropped: `, such as `Dropped: missing price field`. The underlying exception is `scrapy.exceptions.DropItem`. - **Meaning:** An item pipeline deliberately stopped that item from reaching later pipelines and the exporter. - **Likely cause:** A missing required field, a duplicate, an invalid value, or another validation rule you wrote. - **Diagnostic check:** Search your `process_item` methods for `raise DropItem` and compare the reason against the rejected item. - **Smallest fix:** Repair the item or the condition. Do not disable the pipeline to silence the warning. - **Version boundary:** `DropItem` is longstanding pipeline control flow; the reason text varies. Match the filtered host or dropped reason against what you intended to collect. Change the crawl only when there is a real mismatch. ## Why Does scrapy-splash Fail to Import on Scrapy 2.16 and Later? This is a legacy boundary. `scrapy-splash` 0.11.1 imports `scrapy.utils.url.canonicalize_url`, which Scrapy removed in 2.16. The import fails before the crawl starts. ### Legacy scrapy-splash: `ImportError: cannot import name 'canonicalize_url' from 'scrapy.utils.url'` - **Literal message:** Import text is stable. The file path after it varies by environment. - **Meaning:** Loading a spider module imported `scrapy_splash` that failed before the crawl started. - **Likely cause:** `scrapy-splash` 0.11.1 running against Scrapy 2.16 or later. - **Diagnostic check:** Run `python -c "import scrapy, scrapy_splash; print(scrapy.__version__)"` in the active environment. - **Smallest fix:** Do not patch installed package files. `canonicalize_url` now lives in `w3lib.url` but the maintained path is a frozen, already-tested legacy stack or a move to a current browser-rendering integration. - **Version boundary:** Scrapy 2.16.0 removed the deprecated `scrapy.utils.url.canonicalize_url` compatibility import. `scrapy-splash` 0.11.1 imports on Scrapy 2.15 but fails on Scrapy 2.16 and later; the 2.18.0 reproduction raises the exact `ImportError` above. This entry is here because the message still shows up in real upgrades, not because the frozen stack is supported. With all fifteen messages covered, a few recurring questions remain. ## FAQ Is `Filtered offsite request` a Scrapy error?No, not by itself. Scrapy logs it at DEBUG when `OffsiteMiddleware` rejects a host outside `allowed_domains`. Investigate only when that request was supposed to stay in the crawl. Why does Scrapy 2.15 or later show `CannotResolveHostError` instead of Twisted `DNSLookupError`?Scrapy 2.15 added framework-specific download exceptions so handlers expose one consistent error API. Current built-in handlers wrap DNS failures as `CannotResolveHostError`, though older logs still show the underlying Twisted class. Should Scrapy retry `DownloadConnectionRefusedError`?Scrapy includes connection refusal in its default retry exceptions, but a retry only helps when the endpoint is briefly unavailable. A wrong port or stopped local service needs an endpoint fix, not a higher retry count. Does an HTTP 403 prove Scrapy was detected?No. It proves the server refused the request, not why. Keep status and block diagnosis separate from framework exceptions, and read the response body before naming an anti-bot system. ## Summary Route every message by layer before you touch a setting: - **Construction failed.** Inspect the value passed to `Request`. - **Startup failed.** Inspect spider names, dotted component paths, and reactor import order. - **Downloading failed.** Inspect the scheme, then the middleware return contract, then DNS, then host and port. - **The log says offsite or dropped.** Check whether the filter did what you intended. - **The message comes from an old browser integration.** Check the full package matrix before editing framework code. For a clean baseline, see [Web Scraping With Scrapy: The Complete Guide in 2026Build and run a Scrapy 2.18 project with async start(), pagination, selector tests, item validation, pipelines, and JSON Lines export.](https://scrapfly.io/blog/posts/web-scraping-with-scrapy) For the HTTP 403, 429 and challenge responses left out here, [Scrapfly](https://scrapfly.io/products/web-scraping-api) handles that layer separately. ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [How Do You Read a Scrapy Traceback?](#how-do-you-read-a-scrapy-traceback) - [Which Scrapy Request Errors Happen Before a Download?](#which-scrapy-request-errors-happen-before-a-download) - [Scrapy Request: TypeError: Request url must be str, got <type>](#scrapy-request-typeerror-request-url-must-be-str-got-lt-type-gt) - [Scrapy Request: ValueError: Missing scheme in request url: <value>](#scrapy-request-valueerror-missing-scheme-in-request-url-lt-value-gt) - [Scrapy Callback: TypeError: callback must be a callable, got <type>](#scrapy-callback-typeerror-callback-must-be-a-callable-got-lt-type-gt) - [Which Scrapy Component and Twisted Reactor Errors Stop Startup?](#which-scrapy-component-and-twisted-reactor-errors-stop-startup) - [Scrapy SpiderLoader: KeyError: 'Spider not found: <name>'](#scrapy-spiderloader-keyerror-spider-not-found-lt-name-gt) - [Scrapy Component: NameError: Module '<module>' doesn't define any object named '<object>'](#scrapy-component-nameerror-module-lt-module-gt-doesn-t-define-any-object-named-lt-object-gt) - [Scrapy Reactor: RuntimeError: The installed reactor (<actual>) does not match the requested one (<requested>)](#scrapy-reactor-runtimeerror-the-installed-reactor-lt-actual-gt-does-not-match-the-requested-one-lt-requested-gt) - [Twisted Reactor: twisted.internet.error.ReactorNotRestartable](#twisted-reactor-twisted-internet-error-reactornotrestartable) - [Which Scrapy Middleware and Download Handler Errors Stop Requests?](#which-scrapy-middleware-and-download-handler-errors-stop-requests) - [Scrapy NotSupported: Unsupported URL scheme '<scheme>': no handler available for that scheme](#scrapy-notsupported-unsupported-url-scheme-lt-scheme-gt-no-handler-available-for-that-scheme) - [Scrapy Middleware: scrapy.exceptions.\_InvalidOutput: Middleware <class>.process\_response must return Response or Request, got <class 'NoneType'>](#scrapy-middleware-scrapy-exceptions-invalidoutput-middleware-lt-class-gt-process-response-must-return-response-or-request-got-lt-class-nonetype-gt) - [ScrapyDeprecationWarning: <class>.<method>() requires a spider argument](#scrapydeprecationwarning-lt-class-gt-lt-method-gt-requires-a-spider-argument) - [Scrapy DNS: scrapy.exceptions.CannotResolveHostError: DNS lookup failed: <detail>](#scrapy-dns-scrapy-exceptions-cannotresolvehosterror-dns-lookup-failed-lt-detail-gt) - [Scrapy TCP: scrapy.exceptions.DownloadConnectionRefusedError: Connection was refused by other side: <detail>](#scrapy-tcp-scrapy-exceptions-downloadconnectionrefusederror-connection-was-refused-by-other-side-lt-detail-gt) - [Which Scrapy OffsiteMiddleware and DropItem Messages Are Not Failures?](#which-scrapy-offsitemiddleware-and-dropitem-messages-are-not-failures) - [Scrapy OffsiteMiddleware: Filtered offsite request to '<domain>': <GET <url>>](#scrapy-offsitemiddleware-filtered-offsite-request-to-lt-domain-gt-lt-get-lt-url-gt-gt) - [Scrapy DropItem: Dropped: <reason>](#scrapy-dropitem-dropped-lt-reason-gt) - [Why Does scrapy-splash Fail to Import on Scrapy 2.16 and Later?](#why-does-scrapy-splash-fail-to-import-on-scrapy-2-16-and-later) - [Legacy scrapy-splash: ImportError: cannot import name 'canonicalize\_url' from 'scrapy.utils.url'](#legacy-scrapy-splash-importerror-cannot-import-name-canonicalize-url-from-scrapy-utils-url) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python data-parsing ### How to Parse Datetime Strings with Python and Dateparser Dateparser is a popular Python package for parsing datetime strings. Here's how it can be used in web scraping and how t... ](https://scrapfly.io/blog/posts/parsing-datetime-strings-with-python-and-dateparser) [ python data-parsing ### How to Scrape an Entire Product Catalogue with Python Learn how to discover, crawl, and extract every product from an e-commerce catalog in Python, then keep that data fresh ... ](https://scrapfly.io/blog/posts/how-to-scrape-large-product-catalogs) [ http ### Guide to SSL Errors: What do they mean and how to fix them Overview of SSL errors - what are they, what are common issues and how to resolve them. ](https://scrapfly.io/blog/posts/guide-to-ssl-error-meaning-and-fixes) ## Related Questions - [ Q What are Cloudflare Errors 1006, 1007, 1008? ](https://scrapfly.io/blog/answers/cloudflare-error-1006-1007-1008-access-denied) - [ Q How to select elements by class using CSS selectors? ](https://scrapfly.io/blog/answers/how-to-select-elements-by-class-css-selectors) - [ Q How to check if element exists in Playwright? ](https://scrapfly.io/blog/answers/how-to-check-for-element-in-playwright) - [ Q What are scrapy middlewares and how to use them? ](https://scrapfly.io/blog/answers/what-are-scrapy-middlewares-and-how-to-use-them) Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)