# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Unblocker (formerly ASP)](https://scrapfly.io/docs/scrape-api/unblocker) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Schedule](https://scrapfly.io/docs/scrape-api/schedule) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Search](https://scrapfly.io/docs/crawler-api/search) - [Prompt & Extract](https://scrapfly.io/docs/crawler-api/prompt) - [Auto Refresh](https://scrapfly.io/docs/crawler-api/refresh) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Schedule](https://scrapfly.io/docs/crawler-api/schedule) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Schedule](https://scrapfly.io/docs/screenshot-api/schedule) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Data API #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [Captcha Solver](https://scrapfly.io/docs/cloud-browser-api/captcha-solver) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) - [Credential Vault](https://scrapfly.io/docs/cloud-browser-api/credential-vault) - [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp) - [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [1Password](https://scrapfly.io/docs/cloud-browser-api/1password) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [Rust](https://scrapfly.io/docs/sdk/rust) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Javascript Scenario Scrapfly's [js\_scenario](https://scrapfly.io/docs/scrape-api/getting-started?language=ruby#api_param_js_scenario) provides an ability to fully control a headless web browser. Javascript Scenario can be used to issue browser commands like clicking buttons, filling in forms, scrolling and executing custom javascript code. Currently these actions are supported: [ click ](#click) [ fill ](#fill) [ condition ](#condition) [ wait ](#wait) [ scroll ](#scroll) [ execute ](#execute) [ select ](#select) > This feature requires [Javascript Rendering enabled](https://scrapfly.io/docs/scrape-api/javascript-rendering) and the target page must be of HTML type. Javascript scenario details are available in the API response `result.browser_data.js_scenario` as well as the monitoring dashboard: javascript scenario view on monitoring dashboard ## Usage Javascript scenario consists of one or multiple browser actions that are passed to Scrapfly as a [base64 encoded](https://scrapfly.io/web-scraping-tools/base64) JSON array. An average scenario looks something like this: ``` [ {"fill": {"selector": "#username", "value":"demo"}}, {"fill": {"selector": "#password", "value":"demo"}}, {"click": {"selector": "form input[type='submit']"}}, {"wait_for_navigation": {"timeout": 5000}} ] ``` Each scenario step is a JSON object with a single key that represents the action to be performed and details of the action. Once you design your javascript scenario use Scrapfly's [base64 encoding](https://scrapfly.io/web-scraping-tools/base64) online tool to convert it to a base64 encoded string that can be passed to the API for execution. ### Note on Timeouts The entire Javascript Scenario has an execution **budget of 25 seconds**. Scrapfly does a rough estimation on the maximum JS scenario execution time and will reject any scenarios that are estimated to take more than 25 seconds. > For long-running javascript scenario requiring more than 25s - You can check our guide on [how timeout works](https://scrapfly.io/docs/scrape-api/understand-timeout)--- > > **TL;DR** `retry=false` timeout after `60s` by default and you can customize the timeout with `retry=false&timeout=120000` ### Full example with API Player The best way to get familiar with Javascript Scenarios is to use the [Scrapfly Web Player](https://scrapfly.io/dashboard/playground/web-scraper) to design and test your scenario. However, here's an example to get you started. The below scenario will login to web-scraping.dev/login by performing these steps: - Select username input box and fill value `user123` - Select password input box and fill value `password` - Select and click login button - Wait for navigation to acknowledge button click for maximum of 5 seconds ``` [ {"fill": {"selector": "input[name=username]", "value":"user123"}}, {"fill": {"selector": "input[name=password]", "value":"password"}}, {"click": {"selector": "button[type='submit']"}}, {"wait_for_navigation": {"timeout": 5000}} ] ``` Then, this scenario can be base64 encoded and passed to Scrapfly API for execution: [Ruby](#player-85d999) [HTTP](#http-85d999) [ ](https://scrapfly.io/login "Sign in to try from the API player") ``` # gem install httparty require 'httparty' require 'json' require 'base64' # Build query parameters params = { 'render_js' => true, 'js_scenario' => Base64.urlsafe_encode64(JSON.generate([{"fill":{"selector":"input[name=username]","value":"user123"}},{"fill":{"selector":"input[name=password]","value":"password"}},{"click":{"selector":"button[type='submit']"}},{"wait_for_navigation":{"timeout":5000}}])), 'key' => "__API_KEY__", 'url' => "https://web-scraping.dev/login", } url = "https://api.scrapfly.io/scrape" options = { query: params, timeout: 160, open_timeout: 10, } begin response = HTTParty.get(url, options) # Check for HTTP errors unless response.success? error_data = response.parsed_response error_msg = error_data['message'] || error_data['description'] || 'Request failed' raise "HTTP error #{response.code}: #{error_msg}" end data = response.parsed_response puts JSON.pretty_generate(data) # Access the scrape result puts data['result'] if data['result'] rescue HTTParty::Error => e STDERR.puts "Request failed: #{e.message}" raise rescue StandardError => e STDERR.puts "Error: #{e.message}" raise end ``` ``` https://api.scrapfly.io/scrape?render_js=true&js_scenario=W3siZmlsbCI6eyJzZWxlY3RvciI6ImlucHV0W25hbWU9dXNlcm5hbWVdIiwidmFsdWUiOiJ1c2VyMTIzIn19LHsiZmlsbCI6eyJzZWxlY3RvciI6ImlucHV0W25hbWU9cGFzc3dvcmRdIiwidmFsdWUiOiJwYXNzd29yZCJ9fSx7ImNsaWNrIjp7InNlbGVjdG9yIjoiYnV0dG9uW3R5cGU9J3N1Ym1pdCddIn19LHsid2FpdF9mb3JfbmF2aWdhdGlvbiI6eyJ0aW1lb3V0Ijo1MDAwfX1d&key=&url=https%3A%2F%2Fweb-scraping.dev%2Flogin ``` ### Example of response with scenario ``` ... "result": { ..., "browser_data": { "xhr_call": [...], "local_storage_data": { "csm-hit": "tb:s-5B0K136YR4QK89MQ8RG0|1596420691120&t:1596420692684&adb:adblk_no", "csm:adb": "adblk_no", "csm-bf": "[\"5B0K136YR4QK89MQ8RG0\"]", "a-font-class": "a-ember" }, "session_storage_data": { "csm-hit": "tb:s-5B0K136YR4QK89MQ8RG0|1596420691120&t:1596420692684&adb:adblk_no", "csm:adb": "adblk_no", "csm-bf": "[\"5B0K136YR4QK89MQ8RG0\"]", "a-font-class": "a-ember" }, "websockets": [], "javascript_evaluation_result": null, "js_scenario": { "duration": 4.92, "executed": 4, "steps": [ { "action": "fill", "config": { "selector": "input[name=username]", "value": "user123" }, "duration": 1.11, "executed": true, "result": null, "success": true }, { "action": "fill", "config": { "selector": "input[name=password]", "value": "password" }, "duration": 0.47, "executed": true, "result": null, "success": true }, { "action": "click", "config": { "ignore_if_not_visible": false, "selector": "button[type='submit']" }, "duration": 0.52, "executed": true, "result": null, "success": true }, { "action": "wait_for_navigation", "config": { "expect_url": null, "timeout": 5000 }, "duration": 1.81, "executed": true, "result": null, "success": true } ] }, }, ... } ... ``` ## Scenario Step Types Currently, 8 scenario types are supported. Each scenario type has a different set of mandatory and optional parameters. - \[MANDATORY\] param\_name:type - \[OPTIONAL\] param\_name:type ### Click selector:string ignore\_if\_not\_visible:bool=false timeout budget (ms): +2500 Click on a visible element. It's a native click which emits a [trusted event](https://developer.mozilla.org/en-US/docs/Web/API/Event/isTrusted) - **it's not simulated using javascript.** #### Internal Workflow - Waits for the element to be visible - Moves the viewport to the element (mouse and scroll as a human would) - Focuses the element - Left clicks #### Parameters - selector:string Accept CSS Selector and XPATH Selector - ignore\_if\_not\_visible:bool Wait the element if visible or skip if not - multiple:bool If multiple elements match, click on all matched elements #### Usage ``` {"click": {"selector": ".cookie-gdpr-consent", "ignore_if_not_visible": true)}} ``` ``` {"click": {"selector": "submit.btn"}} ``` ### Fill selector:string value:string timeout budget (ms): +${timeout} +500Type a text value in the targeted element. The typing is not simulated using javascript - it's from a real keyboard input. #### Internal Workflow - Waits for the element to be visible - Moves the viewport to the element (mouse and scroll as a human would) - Focuses the element - Types the text value into the input as a human would #### Parameters - selector:string Any **valid** CSS and XPATH Selector - value:string Value to type in element - clear:boolean Clear the input field before writing #### Usage ``` {"fill": {"selector": "#name", "value": "John Do")}} ``` ### Condition Condition are exclusive to one of - status\_code:int - selector:string - selector\_state:string=existing - timeout:int=1000 action:string=continue### Parameters - selector:string Any **valid** CSS or XPATH selector - selector\_state:string Can be `existing` or `not_existing` - action:string Action when the condition is met, can be `continue`, `exit_success`, `exit_failed` Play the scenario only if the condition is met. For example, it can stop processing scenario if non-200 status code is encountered. #### Internal Workflow - Checks whether response status code matches the required status code codintions #### Usage ``` {"condition": {"status_code": 200}} ``` ### Wait timeout budget (ms): +${wait}Pause during the scenario to give browser some time to load. Note that the pause time is part of the scenario budget #### Parameters There are no parameters - pass the wait time in milliseconds directly #### Usage ``` {"wait": 2000} ``` ### Scroll element:string=body selector:string=bottom timeout budget (ms): +500 Scroll to the selector (or the bottom of the page if no selector provided). If the element parameter is a valid selector, the scrolling wil be executed within the selected element. The scroll is not simulated using javascript - it's created with a real mouse input. #### Internal Workflow - Waits for the selector to be visible (if set) - Waits for the element to be visible (if set) and binds scroll to the element - Scrolls the page as a human would #### Parameters - element:string=body a valid css selector or xpath or "body" - selector:string a valid css selector or xpath or "bottom" as target to scroll - infinite:int=0 infinite scroll - number of scroll iteration - click\_selector:string a valid css selector or xpath to click on after the scroll - like a "view more" button #### Usage ``` {"scroll": {"selector": "bottom"}} ``` ``` {"scroll": {"selector": "#pricing"}} ``` ``` {"scroll": {"element": "#scrollable-list", "selector": "bottom", "infinite": 2}} ``` ### Execute timeout:int=3000 timeout budget (ms): +${timeout}Execute a javascript script and store the result if a result is returned #### Internal Workflow - The Javascript code is executed - If the javascript code returns a value - it's stored and available in API response `result.browser_data.js_scenario.steps`. Note that each "execute" step has a `result` entry. - Supports async/await function #### Parameters - script:string Script to execute. It can return a serializable value - timeout:int Timeout to wait after the script execution have started - expressed in millisecond #### Usage ``` {"execute": {"script": "document.querySelector(\"body\").style.backgroundColor = \"red\";}"} ``` ``` {"execute": {"script": "return navigator.userAgent", "timeout": "1000"} ``` ### Wait For Navigation timeout:int=1000 timeout budget (ms): +${timeout} + 1500 Time to wait to detect a navigation / changing page. The given timeout + 1500 (1.5s) is added to the scenario budget - this additional time represent the average duration of a standard page loading (with assets, xhr, etc). For example if you set a timeout of 1000, 2500 is counted. #### Parameters - timeout:int Maximum timeout to wait for a navigation - expressed in millisecond #### Usage ``` {"wait_for_navigation": {}} ``` ``` {"wait_for_navigation": {"timeout": 5000}} ``` ### Wait For Selector selector:string=body state:string=visible ignore:bool=false timeout budget (ms): +${timeout} Wait the element is visible (if state=visible) in the page or the element disappear (state=hidden). If the selector is not present in the desired state until the timeout this step failed and the scenario is aborted, unless ignore=true is set - the failure is then suppressed and the scenario continues with the next step. The timeout is added to the scenario budget #### Parameters - selector:string=body a valid css selector or xpath or "body" - state:string=visible state of the element in the page "visible" or "hidden" - timeout:int=5000 Timeout to wait before fail - expressed in milliseconds - ignore:bool=false Continue with the next step instead of aborting the scenario when the selector never reaches the state before the timeout. The wait is still performed, so the timeout is spent before the step gives up. An invalid selector is silenced as well, so a typo in the selector becomes a no-op instead of an error #### Usage ``` {"wait_for_selector": {"selector": "#pricing"}} ``` ``` {"wait_for_selector": {"selector": "#loading", "state": "hidden", "timeout": 10000}} ``` ``` {"wait_for_selector": {"selector": ".cookie-gdpr-consent", "timeout": 3000, "ignore": true}} ``` ### Select selector:string value:string option\_selector:string text:string index:int timeout:int=5000 timeout budget (ms): +${timeout} Select an option from a dropdown menu. Supports both native HTML ``: selects the option matching the given value - For custom dropdowns: clicks the trigger, waits for options to appear, then clicks the matching option #### Parameters - selector:string CSS or XPath selector for the dropdown trigger element - value:string For native `` elements, or `option_selector` for custom dropdowns. When using `option_selector`, optionally add `text` or `index` to narrow the match. #### Usage **Native `