# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Schedule](https://scrapfly.io/docs/scrape-api/schedule) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Schedule](https://scrapfly.io/docs/crawler-api/schedule) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Schedule](https://scrapfly.io/docs/screenshot-api/schedule) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Data API #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [Captcha Solver](https://scrapfly.io/docs/cloud-browser-api/captcha-solver) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Session Scrapfly's [session](https://scrapfly.io/docs/scrape-api/getting-started?view=markdown#api_param_session) feature allows for a consistent navigation profile to be kept across multiple scrape requests. In other words, requests with the same `session` key will share the same cookies, referrer, and navigation history and proxy (unless [session\_sticky\_proxy](https://scrapfly.io/docs/scrape-api/getting-started?view=markdown#api_param_session_sticky_proxy)) is disabled). Sessions can be shared between scrape requests with [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering?view=markdown) and without it. So, you can mix navigation and benefit from a better performance by avoiding JavaScript rendering when it's not necessary. If JavaScript rendering is activated, it also persists and restores `window.localStorage` and `window.sessionStorage` data. Session use can increase the success rate of your scraping jobs when scraping anti-bot protected websites. When **[Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection)** auto activates, it configures the session with all the bypass details making subsequent, same-session requests more likely to succeed. When inspecting the log from the web interface, you can see all the session details such as navigation history, stored cookies, referrer, and all the metadata-creation date, last used date, expiration date: session details on the monitoring logs page ## Sharing Policy Sessions are **not** shared across Scrapfly [projects](https://scrapfly.io/docs/project?view=markdown), environments or scrapfly team members. Sessions **can be shared** across scrape requests with [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering?view=markdown) and without it. So, you can use Javascript Rendering to establish state cookies and continue without it speeding up your scraping jobs significantly. > Note to name your session with a unique name in your Scrapfly project/environment. We recommend generating UUID4 to prevent accidental name overlap. ## Eviction Policy A session automatically **expires after seven days**. Each time a session is reused, the expiration countdown is reset back to seven days. A session sticks to a proxy by default (unless [session\_sticky\_proxy](https://scrapfly.io/docs/scrape-api/getting-started?view=markdown#api_param_session_sticky_proxy)) is disabled). After 30 seconds of session inactivity, the proxy IP is not guaranteed and a new one might be assigned. This entirely depends on the used proxy pool as some proxy pools, like the residential ones, have a low lifetime. ## Limitation Session feature cannot be used while [Cache](https://scrapfly.io/docs/scrape-api/cache?view=markdown) is activated. ## Usage - [Session](#session) - [Session without sticky Proxy](#sticky_proxy) For example, this scrape requests simulate website setting cookies and we set our session as `mysession1` ``` curl -G \ --request "GET" \ --url "https://api.scrapfly.io/scrape" \ --data-urlencode "session=mysession1" \ --data-urlencode "key=__API_KEY__" \ --data-urlencode "url=https://httpbin.dev/cookies/set?mycookie=123" ``` Then, if we continue with more scrapfly requests with the same session the cookies will persist automatically: ``` curl -G \ --request "GET" \ --url "https://api.scrapfly.io/scrape" \ --data-urlencode "session=mysession1" \ --data-urlencode "key=__API_KEY__" \ --data-urlencode "url=https://httpbin.dev/cookies" ``` > This feature is not recommended as a natural way of browsing with sessions is using a sticky proxy for all related requests. With `session_sticky_proxy` disabled, the proxy IP will rotate for each request making the connection traffic appear very unnatural. Use at your own risk! > When `asp` is **enabled**, `session_sticky_proxy` is ignored during an ASP rotation: if the bound exit IP gets blocked, ASP is free to rotate to a fresh IP to re-solve the challenge rather than staying pinned to a dead one. Sticky behavior still applies to the steady-state session traffic between rotations. ``` curl -G \ --request "GET" \ --url "https://api.scrapfly.io/scrape" \ --data-urlencode "session=mysession1" \ --data-urlencode "session_sticky_proxy=false" \ --data-urlencode "key=__API_KEY__" \ --data-urlencode "url=https://httpbin.dev/cookies/set?mycookie=123" ``` #### Example Of Response ``` ... "context": { ... "session": { "name": "mysession1", "state": "FREE", "lease": null, "correlation_id": "default", "identity": "929f7fb918367df788717d825a3d75391e148c76", "created_at": "2020-09-15 12:44:15 UTC", "cookie_jar": [ { "comment": null, "domain": "httpbin.dev", "expires": null, "http_only": false, "max_age": null, "name": "mycookie", "path": "/cookies", "secure": false, "size": null, "value": "123", "version": 0 }, ], "last_used_at": "2023-09-15 12:44:50 UTC", "expire_at": "2023-09-22 12:44:50 UTC", "referer": "https://httpbin.dev/" } ... } ... ``` ## Related Errors All related errors are listed below. You can see full description and example of error response on [Errors section](https://scrapfly.io/docs/scrape-api/errors#proxy) - [ERR::SESSION::CLOUD\_BROWSER\_NOT\_FOUND](https://scrapfly.io/docs/scrape-api/error/ERR::SESSION::CLOUD_BROWSER_NOT_FOUND "A session prefixed with `cb:` was requested but no Cloud Browser profile is available for handoff. Ensure the Cloud Browser run completed and the session was stopped gracefully before calling the Web Scraping API.") - A session prefixed with `cb:` was requested but no Cloud Browser profile is available for handoff. Ensure the Cloud Browser run completed and the session was stopped gracefully before calling the Web Scraping API. - [ERR::SESSION::CONCURRENT\_ACCESS](https://scrapfly.io/docs/scrape-api/error/ERR::SESSION::CONCURRENT_ACCESS "Concurrent access to the session has been tried. If your spider run on distributed architecture, the same session name is currently used by another scrape") - Concurrent access to the session has been tried. If your spider run on distributed architecture, the same session name is currently used by another scrape - [ERR::SESSION::PROXY\_POOL\_MISMATCH](https://scrapfly.io/docs/scrape-api/error/ERR::SESSION::PROXY_POOL_MISMATCH "Session was created with a different proxy pool than requested. Sessions are bound to their original proxy pool and cannot switch between datacenter/residential pools.") - Session was created with a different proxy pool than requested. Sessions are bound to their original proxy pool and cannot switch between datacenter/residential pools. ## Cloud Browser Session A session ID prefixed with `cb:` reuses the state of a Cloud Browser run that you previously stopped. The typical use case is to handle login, MFA, or JS challenges in [Cloud Browser](https://scrapfly.io/docs/cloud-browser-api/getting-started), stop the session, then continue with regular Web Scraping API requests against the same target. The Cloud Browser side of this flow is covered on the [Cloud Browser Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume#web-scraping-api-handoff) page. > Format: `session=cb:`. The first matching request reads the Cloud Browser profile from storage, converts it into a standard Web Scraping API session, and persists it. Subsequent requests with the same session ID use the persisted session directly. The Cloud Browser profile is read once. ### What is inherited from the Cloud Browser - **Cookies**: all cookies the browser stored, including HttpOnly and Secure flags. - **Local Storage**: per-origin key/value pairs from `window.localStorage`. - **Session Storage**: per-origin key/value pairs from `window.sessionStorage`. - **Private State Tokens** (Trust Tokens): issued tokens, redemption records, issuer associations. - **Exit IP and proxy identity**: same upstream proxy pool, country, and sticky-session credentials. Within the upstream provider's sticky window, follow-up requests come out of the same exit IP. - **Fingerprint**: rebuilt from the Cloud Browser seed. Browser brand, platform, locale, language, and timezone match what the site originally saw. > **One-way handoff.** The Web Scraping API session is a snapshot taken when the Cloud Browser run was stopped. It does not sync back to the Cloud Browser profile. Cookie, storage, or token changes made by the Web Scraping API stay in the Web Scraping API session. ### Usage Stop the Cloud Browser session you authenticated in, then call the Web Scraping API with `session=cb:`: ``` curl "https://api.scrapfly.io/scrape?key=&url=https://example.com/account&session=cb:my-cloud-browser-session" ``` Reuse the same session name on subsequent requests. Cookies, storage, fingerprint, and exit IP follow each request like any other session. ## Integration - [Session example with Python SDK](https://scrapfly.io/docs/onboarding#session) - [Cloud Browser API documentation](https://scrapfly.io/docs/cloud-browser-api/getting-started)