[Answers](https://scrapfly.io/blog) / [http](https://scrapfly.io/blog/tag/http) / [How to Persist Sessions Across Scrapy Requests](https://scrapfly.io/blog/answers/persist-sessions-across-scrapy-requests) # How to Persist Sessions Across Scrapy Requests by [Mohab Yousry](https://scrapfly.io/blog/author/mohab-yousry-9396552a) Sep 29, 2026 5 min read [\#http](https://scrapfly.io/blog/tag/http) [\#scrapy](https://scrapfly.io/blog/tag/scrapy) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fpersist-sessions-across-scrapy-requests "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fpersist-sessions-across-scrapy-requests&text=How%20to%20Persist%20Sessions%20Across%20Scrapy%20Requests "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fpersist-sessions-across-scrapy-requests "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fpersist-sessions-across-scrapy-requests) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fpersist-sessions-across-scrapy-requests) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fpersist-sessions-across-scrapy-requests) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fpersist-sessions-across-scrapy-requests) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fanswers%2Fpersist-sessions-across-scrapy-requests) The Scrapy framework stores the cookies automatically between requests via CookiesMiddleware; however, it is only valid within one run of the spider. You can use the `meta["cookiejar"]` feature to keep different sessions separate, handle reauthentication and seeding by yourself after a restart, and load a Playwright `storage_state` file. ## What Does a Scrapy Session Persist? Scrapy has no single session object. Each primitive below keeps different state, and each stops at a different boundary. | State primitive | What it keeps | Boundary | Use it when | |---|---|---|---| | CookiesMiddleware default jar | Valid server cookies | Current spider run | One HTTP session is enough | | `meta["cookiejar"]` | Separate cookie set per identifier | Current spider run | Accounts or searches must not share cookies | | `Request.cookies` or auth header | Explicit state supplied by your code | Each request or startup path | State comes from a secure store or login step | | JOBDIR plus `spider.state` | Queue, seen requests, simple values | Clean pause/resume on the same Scrapy version | A crawl must resume where it stopped | | Playwright browser context | Cookies and browser-side storage | Context lifetime or exported storage state | Authentication depends on JavaScript storage | | Cloud Browser Session Resume | Managed browser state across reconnects | Stable session ID and retained remote session | The browser process itself must resume | Cookie continuity is not network continuity. Two requests can share cookies and still leave from different proxy IPs or TLS fingerprints, so keep those stable separately if the target ties the session to them. For cookie basics, see [how to handle cookies in web scraping](https://scrapfly.io/blog/posts/how-to-handle-cookies-in-web-scraping). ## How Does CookiesMiddleware Persist Cookies Across Scrapy Requests? CookiesMiddleware is enabled by default. It stores every `Set-Cookie` value and attaches the valid cookies to later requests for the rest of the spider run. python```python import scrapy class SessionCookiesSpider(scrapy.Spider): name = "session_cookies" async def start(self): # The server replies with Set-Cookie and a redirect yield scrapy.Request( "https://httpbin.dev/cookies/set?session=scrapy", callback=self.after_cookie, ) def after_cookie(self, response): # No manual cookie handling, CookiesMiddleware attaches the jar yield scrapy.Request( "https://httpbin.dev/cookies?step=2", callback=self.parse_cookies, ) def parse_cookies(self, response): cookies = response.json() yield {"session": cookies["session"], "requests": 2} ``` json```json [ {"session": "scrapy", "requests": 2} ] ``` The second request never set any cookies, and `/cookies` shows `session=scrapy`. In order to set a cookie manually, use `Request.cookies={"name": "value"}`. Do not use `Cookie` header directly because CookiesMiddleware drops this header and thus the value never gets to the server. Use `Request.cookies` unless you disable CookiesMiddleware. Setting `COOKIES_DEBUG=True` will log all sent and received cookies. This should be used locally only due to security reasons. ## How Do Multiple Scrapy Cookie Jars Isolate Sessions? Give each session a stable `cookiejar` identifier in `meta` and pass that same identifier on every request in the chain. python```python import scrapy class CookieJarsSpider(scrapy.Spider): name = "cookie_jars" async def start(self): for jar in ("alpha", "beta"): yield scrapy.Request( f"https://httpbin.dev/cookies/set?account={jar}", meta={"cookiejar": jar}, dont_filter=True, ) def parse(self, response): # cookiejar is not sticky, so pass it on every follow-up request yield scrapy.Request( "https://httpbin.dev/cookies?step=2", meta={"cookiejar": response.meta["cookiejar"]}, callback=self.parse_jar, dont_filter=True, ) def parse_jar(self, response): yield {"jar": response.meta["cookiejar"], "cookies": response.json()} ``` Scrapfly #### Scale your web scraping effortlessly Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data. [Try Free →](https://scrapfly.io/register)json```json [ {"jar": "beta", "cookies": {"account": "beta"}}, {"jar": "alpha", "cookies": {"account": "alpha"}} ] ``` Both sessions run concurrently and neither sees the other's cookie. The `cookiejar` key is not sticky, so a follow-up request without it falls back to the default jar. `cookies={}` seeds values into a jar, while `cookiejar` selects which jar to use. A cookie jar also does not pin the proxy IP, so assign proxies separately as shown in [Scrapy proxy rotation](https://scrapfly.io/blog/answers/scrapy-spiders-proxy-rotation). ## What Survives a Scrapy Restart With JOBDIR? JOBDIR restores the crawl queue, the duplicate filter, and simple `spider.state` values after a clean pause. Cookies from the previous run are gone. shell```shell scrapy crawl catalog -s JOBDIR=crawls/catalog-1 ``` Stop crawling cleanly, make sure that you have only one directory per job, continue on the same Scrapy version and maintain request serializability. As the jar is empty when resuming crawling, you have to check the session and authenticate yourself once more, or set your cookies/tokens securely and validate them. Scrapy [issue #5930](https://github.com/scrapy/scrapy/issues/5930) stopped built-in cross run cookie persistence as being not planned. ## How Does scrapy-playwright Persist Browser Authentication? scrapy-playwright keeps state inside a named browser context while it stays open. Across processes, load a Playwright [storage\_state](https://playwright.dev/python/docs/auth) file into a new context. `storage_state` restores cookies and localStorage. IndexedDB needs `indexed_db=True` when exporting, and sessionStorage is not saved at all. The file can impersonate the account, so keep it out of Git. python```python import scrapy class AuthStateSpider(scrapy.Spider): name = "auth_state" custom_settings = { "DOWNLOAD_HANDLERS": { "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", }, } async def start(self): yield scrapy.Request( "https://web-scraping.dev/login", meta={ "playwright": True, "playwright_context": "account", "playwright_context_kwargs": { "storage_state": "playwright/.auth/state.json", }, }, ) def parse(self, response): yield {"status": response.css("div.form-text::text").get("").strip()} ``` json```json [ {"status": "Logged in as User123"} ] ``` The context opened already logged in. `playwright_context_kwargs` is only read when the named context is created, and Playwright cookies are not shared with CookiesMiddleware. For setup, see the [Scrapy Playwright tutorial](https://scrapfly.io/blog/posts/how-to-use-scrapy-with-playwright). To keep sessionStorage too, [Cloud Browser Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) reconnects to the same managed browser with the same `session` value and `auto_close=false`. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [What Does a Scrapy Session Persist?](#what-does-a-scrapy-session-persist) - [How Does CookiesMiddleware Persist Cookies Across Scrapy Requests?](#how-does-cookiesmiddleware-persist-cookies-across-scrapy-requests) - [How Do Multiple Scrapy Cookie Jars Isolate Sessions?](#how-do-multiple-scrapy-cookie-jars-isolate-sessions) - [What Survives a Scrapy Restart With JOBDIR?](#what-survives-a-scrapy-restart-with-jobdir) - [How Does scrapy-playwright Persist Browser Authentication?](#how-does-scrapy-playwright-persist-browser-authentication) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ http ### How to Handle Cookies in Web Scraping Introduction to cookies in web scraping. What are they and how to take advantage of cookie process to authenticate or se... ](https://scrapfly.io/blog/posts/how-to-handle-cookies-in-web-scraping) [ http python ### Guide to Python requests POST method Discover how to use Python's requests library for POST requests, including JSON, form data, and file uploads, along with... ](https://scrapfly.io/blog/posts/how-to-python-requests-post) [ python blocking ### 9 Mechanisms to Check When Your Scrapy Spider Gets Blocked in 2026 Diagnostic walkthrough for a blocked Scrapy spider, covering IP reputation, rate shape, cookie and session state, header... ](https://scrapfly.io/blog/posts/why-scrapy-spider-gets-blocked) ## Related Questions - [ Q How to save and load cookies in Python requests? ](https://scrapfly.io/blog/answers/save-and-load-cookies-in-requests-python) - [ Q How to save and load cookies in Playwright? ](https://scrapfly.io/blog/answers/how-to-save-and-load-cookies-in-playwright) - [ Q How to Set cURL Authentication - Full Examples Guide ](https://scrapfly.io/blog/answers/how-to-set-authorization-with-curl-full-examples-guide) - [ Q How to save and load cookies in Puppeteer? ](https://scrapfly.io/blog/answers/how-to-save-and-load-cookies-in-puppeteer) Scale your web scraping effortlessly, **1,000 free credits** [Start Free](https://scrapfly.io/register)