# Scrapfly Documentation ## Table of Contents ### Dashboard - [Intro](https://scrapfly.io/docs) - [Project](https://scrapfly.io/docs/project) - [Account](https://scrapfly.io/docs/account) - [Workspace & Team](https://scrapfly.io/docs/workspace-and-team) - [Billing](https://scrapfly.io/docs/billing) ### Products #### MCP Server - [Getting Started](https://scrapfly.io/docs/mcp/getting-started) - [Tools & API Spec](https://scrapfly.io/docs/mcp/tools) - [Authentication](https://scrapfly.io/docs/mcp/authentication) - [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples) - [FAQ](https://scrapfly.io/docs/mcp/faq) ##### Integrations - [Overview](https://scrapfly.io/docs/mcp/integrations) - [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop) - [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code) - [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt) - [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor) - [Cline](https://scrapfly.io/docs/mcp/integrations/cline) - [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf) - [Zed](https://scrapfly.io/docs/mcp/integrations/zed) - [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code) - [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode) - [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain) - [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex) - [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai) - [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai) - [n8n](https://scrapfly.io/docs/mcp/integrations/n8n) - [Make](https://scrapfly.io/docs/mcp/integrations/make) - [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier) - [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi) - [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder) - [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client) #### Web Scraping API - [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started) - [API Specification]() - [Monitoring](https://scrapfly.io/docs/monitoring) - [Customize Request](https://scrapfly.io/docs/scrape-api/custom) - [Debug](https://scrapfly.io/docs/scrape-api/debug) - [Anti Scraping Protection](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - [Proxy](https://scrapfly.io/docs/scrape-api/proxy) - [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode) - [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog) - [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify) - [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction) - [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario) - [SSL](https://scrapfly.io/docs/scrape-api/ssl) - [DNS](https://scrapfly.io/docs/scrape-api/dns) - [Cache](https://scrapfly.io/docs/scrape-api/cache) - [Session](https://scrapfly.io/docs/scrape-api/session) - [Webhook](https://scrapfly.io/docs/scrape-api/webhook) - [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot) - [Errors](https://scrapfly.io/docs/scrape-api/errors) - [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout) - [Throttling](https://scrapfly.io/docs/throttling) - [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot) - [Billing](https://scrapfly.io/docs/scrape-api/billing) - [FAQ](https://scrapfly.io/docs/scrape-api/faq) #### Crawler API - [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started) - [API Specification]() - [Retrieving Results](https://scrapfly.io/docs/crawler-api/results) - [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format) - [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules) - [Webhook](https://scrapfly.io/docs/crawler-api/webhook) - [Billing](https://scrapfly.io/docs/crawler-api/billing) - [Errors](https://scrapfly.io/docs/crawler-api/errors) - [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot) - [FAQ](https://scrapfly.io/docs/crawler-api/faq) #### Screenshot API - [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started) - [API Specification]() - [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility) - [Webhook](https://scrapfly.io/docs/screenshot-api/webhook) - [Billing](https://scrapfly.io/docs/screenshot-api/billing) - [Errors](https://scrapfly.io/docs/screenshot-api/errors) #### Extraction API - [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started) - [API Specification]() - [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template) - [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt) - [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai) - [Webhook](https://scrapfly.io/docs/extraction-api/webhook) - [Billing](https://scrapfly.io/docs/extraction-api/billing) - [Errors](https://scrapfly.io/docs/extraction-api/errors) - [FAQ](https://scrapfly.io/docs/extraction-api/faq) #### Proxy Saver - [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started) - [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints) - [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations) - [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates) - [Protocols](https://scrapfly.io/docs/proxy-saver/protocols) - [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile) - [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security) - [Billing](https://scrapfly.io/docs/proxy-saver/billing) #### Cloud Browser API - [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started) - [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy) - [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock) - [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads) - [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume) - [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) - [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode) - [Bring Your Own Proxy](https://scrapfly.io/docs/cloud-browser-api/bring-your-own-proxy) - [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions) ##### Integrations - [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer) - [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright) - [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium) - [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser) - [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use) - [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand) - [Billing](https://scrapfly.io/docs/cloud-browser-api/billing) - [Errors](https://scrapfly.io/docs/cloud-browser-api/errors) ### Tools - [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector) ### SDK - [Golang](https://scrapfly.io/docs/sdk/golang) - [Python](https://scrapfly.io/docs/sdk/python) - [TypeScript](https://scrapfly.io/docs/sdk/typescript) - [Scrapy](https://scrapfly.io/docs/sdk/scrapy) ### Integrations - [Getting Started](https://scrapfly.io/docs/integration/getting-started) - [LangChain](https://scrapfly.io/docs/integration/langchain) - [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex) - [CrewAI](https://scrapfly.io/docs/integration/crewai) - [Zapier](https://scrapfly.io/docs/integration/zapier) - [Make](https://scrapfly.io/docs/integration/make) - [n8n](https://scrapfly.io/docs/integration/n8n) ### Academy - [Overview](https://scrapfly.io/academy) - [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview) - [Tools](https://scrapfly.io/academy/tools-overview) - [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering) - [Static Scraping](https://scrapfly.io/academy/static-scraping) - [HTML Parsing](https://scrapfly.io/academy/html-parsing) - [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping) - [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping) - [Headless Browsers](https://scrapfly.io/academy/headless-browsers) - [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data) - [JSON Parsing](https://scrapfly.io/academy/json-parsing) - [Data Processing](https://scrapfly.io/academy/data-processing) - [Scaling](https://scrapfly.io/academy/scaling) - [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary) - [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking) - [Proxies](https://scrapfly.io/academy/proxies) --- # Javascript Scenario [ View as markdown ](https://scrapfly.io/?view=markdown) Copy for LLM Copy for LLM [ Open in ChatGPT ](https://chatgpt.com/?hints=search&prompt=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fscrape-api%2Fjavascript-scenario%3Flanguage%3Druby%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Claude ](https://claude.ai/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fscrape-api%2Fjavascript-scenario%3Flanguage%3Druby%20so%20I%20can%20ask%20questions%20about%20it.) [ Open in Perplexity ](https://www.perplexity.ai/search/new?q=Read%20from%20https%3A%2F%2Fscrapfly.io%2Fdocs%2Fscrape-api%2Fjavascript-scenario%3Flanguage%3Druby%20so%20I%20can%20ask%20questions%20about%20it.) Scrapfly's [js\_scenario](https://scrapfly.io/docs/scrape-api/getting-started?language=ruby#api_param_js_scenario) provides an ability to fully control a headless web browser. Javascript Scenario can be used to issue browser commands like clicking buttons, filling in forms, scrolling and executing custom javascript code. Currently these actions are supported: [ click ](#click) [ fill ](#fill) [ condition ](#condition) [ wait ](#wait) [ scroll ](#scroll) [ execute ](#execute) [ select ](#select) > This feature requires [Javascript Rendering enabled](https://scrapfly.io/docs/scrape-api/javascript-rendering) and the target page must be of HTML type. Javascript scenario details are available in the API response `result.browser_data.js_scenario` as well as the monitoring dashboard: javascript scenario view on monitoring dashboard ## Usage Javascript scenario consists of one or multiple browser actions that are passed to Scrapfly as a [base64 encoded](https://scrapfly.io/web-scraping-tools/base64) JSON array. An average scenario looks something like this: ``` [ {"fill": {"selector": "#username", "value":"demo"}}, {"fill": {"selector": "#password", "value":"demo"}}, {"click": {"selector": "form input[type='submit']"}}, {"wait_for_navigation": {"timeout": 5000}} ] ``` Each scenario step is a JSON object with a single key that represents the action to be performed and details of the action. Once you design your javascript scenario use Scrapfly's [base64 encoding](https://scrapfly.io/web-scraping-tools/base64) online tool to convert it to a base64 encoded string that can be passed to the API for execution. ### Note on Timeouts The entire Javascript Scenario has an execution **budget of 25 seconds**. Scrapfly does a rough estimation on the maximum JS scenario execution time and will reject any scenarios that are estimated to take more than 25 seconds. > For long-running javascript scenario requiring more than 25s - You can check our guide on [how timeout works](https://scrapfly.io/docs/scrape-api/understand-timeout)--- > > **TL;DR** `retry=false` timeout after `90s` by default and you can customize the timeout with `retry=false&timeout=120000` ### Full example with API Player The best way to get familiar with Javascript Scenarios is to use the [Scrapfly Web Player](https://scrapfly.io/dashboard/playground/web-scraper) to design and test your scenario. However, here's an example to get you started. The below scenario will login to web-scraping.dev/login by performing these steps: - Select username input box and fill value `user123` - Select password input box and fill value `password` - Select and click login button - Wait for navigation to acknowledge button click for maximum of 5 seconds ``` [ {"fill": {"selector": "input[name=username]", "value":"user123"}}, {"fill": {"selector": "input[name=password]", "value":"password"}}, {"click": {"selector": "button[type='submit']"}}, {"wait_for_navigation": {"timeout": 5000}} ] ``` Then, this scenario can be base64 encoded and passed to Scrapfly API for execution: [Ruby](#player-7f5f8d) [HTTP](#http-7f5f8d) [ ](https://scrapfly.io/login "Sign in to try from the API player") ``` # gem install httparty require 'httparty' require 'json' # Build query parameters params = { 'render_js' => true, 'js_scenario' => "W3siZmlsbCI6eyJzZWxlY3RvciI6ImlucHV0W25hbWU9dXNlcm5hbWVdIiwidmFsdWUiOiJ1c2VyMTIzIn19LHsiZmlsbCI6eyJzZWxlY3RvciI6ImlucHV0W25hbWU9cGFzc3dvcmRdIiwidmFsdWUiOiJwYXNzd29yZCJ9fSx7ImNsaWNrIjp7InNlbGVjdG9yIjoiYnV0dG9uW3R5cGU9J3N1Ym1pdCddIn19LHsid2FpdF9mb3JfbmF2aWdhdGlvbiI6eyJ0aW1lb3V0Ijo1MDAwfX1d", 'key' => "__API_KEY__", 'url' => "https://web-scraping.dev/login", } url = "https://api.scrapfly.io/scrape" options = { query: params, timeout: 160, open_timeout: 10, } begin response = HTTParty.get(url, options) # Check for HTTP errors unless response.success? error_data = response.parsed_response error_msg = error_data['message'] || error_data['description'] || 'Request failed' raise "HTTP error #{response.code}: #{error_msg}" end data = response.parsed_response puts JSON.pretty_generate(data) # Access the scrape result puts data['result'] if data['result'] rescue HTTParty::Error => e STDERR.puts "Request failed: #{e.message}" raise rescue StandardError => e STDERR.puts "Error: #{e.message}" raise end ``` ``` https://api.scrapfly.io/scrape?render_js=true&js_scenario=W3siZmlsbCI6eyJzZWxlY3RvciI6ImlucHV0W25hbWU9dXNlcm5hbWVdIiwidmFsdWUiOiJ1c2VyMTIzIn19LHsiZmlsbCI6eyJzZWxlY3RvciI6ImlucHV0W25hbWU9cGFzc3dvcmRdIiwidmFsdWUiOiJwYXNzd29yZCJ9fSx7ImNsaWNrIjp7InNlbGVjdG9yIjoiYnV0dG9uW3R5cGU9J3N1Ym1pdCddIn19LHsid2FpdF9mb3JfbmF2aWdhdGlvbiI6eyJ0aW1lb3V0Ijo1MDAwfX1d&key=&url=https%3A%2F%2Fweb-scraping.dev%2Flogin ``` ### Example of response with scenario ``` ... "result": { ..., "browser_data": { "xhr_call": [...], "local_storage_data": { "csm-hit": "tb:s-5B0K136YR4QK89MQ8RG0|1596420691120&t:1596420692684&adb:adblk_no", "csm:adb": "adblk_no", "csm-bf": "[\"5B0K136YR4QK89MQ8RG0\"]", "a-font-class": "a-ember" }, "session_storage_data": { "csm-hit": "tb:s-5B0K136YR4QK89MQ8RG0|1596420691120&t:1596420692684&adb:adblk_no", "csm:adb": "adblk_no", "csm-bf": "[\"5B0K136YR4QK89MQ8RG0\"]", "a-font-class": "a-ember" }, "websockets": [], "javascript_evaluation_result": null, "js_scenario": { "duration": 4.92, "executed": 4, "steps": [ { "action": "fill", "config": { "selector": "input[name=username]", "value": "user123" }, "duration": 1.11, "executed": true, "result": null, "success": true }, { "action": "fill", "config": { "selector": "input[name=password]", "value": "password" }, "duration": 0.47, "executed": true, "result": null, "success": true }, { "action": "click", "config": { "ignore_if_not_visible": false, "selector": "button[type='submit']" }, "duration": 0.52, "executed": true, "result": null, "success": true }, { "action": "wait_for_navigation", "config": { "expect_url": null, "timeout": 5000 }, "duration": 1.81, "executed": true, "result": null, "success": true } ] }, }, ... } ... ``` ## Scenario Step Types Currently, 8 scenario types are supported. Each scenario type has a different set of mandatory and optional parameters. - \[MANDATORY\] param\_name:type - \[OPTIONAL\] param\_name:type ### Click selector:string ignore\_if\_not\_visible:bool=false timeout budget (ms): +2500 Click on a visible element. It's a native click which emits a [trusted event](https://developer.mozilla.org/en-US/docs/Web/API/Event/isTrusted) - **it's not simulated using javascript.** #### Internal Workflow - Waits for the element to be visible - Moves the viewport to the element (mouse and scroll as a human would) - Focuses the element - Left clicks #### Parameters - selector:string Accept CSS Selector and XPATH Selector - ignore\_if\_not\_visible:bool Wait the element if visible or skip if not - multiple:bool If multiple elements match, click on all matched elements #### Usage ``` {"click": {"selector": ".cookie-gdpr-consent", "ignore_if_not_visible": true)}} ``` ``` {"click": {"selector": "submit.btn"}} ``` ### Fill selector:string value:string timeout budget (ms): +${timeout} +500Type a text value in the targeted element. The typing is not simulated using javascript - it's from a real keyboard input. #### Internal Workflow - Waits for the element to be visible - Moves the viewport to the element (mouse and scroll as a human would) - Focuses the element - Types the text value into the input as a human would #### Parameters - selector:string Any **valid** CSS and XPATH Selector - value:string Value to type in element - clear:boolean Clear the input field before writing #### Usage ``` {"fill": {"selector": "#name", "value": "John Do")}} ``` ### Condition Condition are exclusive to one of - status\_code:int - selector:string - selector\_state:string=existing - timeout:int=1000 action:string=continue### Parameters - selector:string Any **valid** CSS or XPATH selector - selector\_state:string Can be `existing` or `not_existing` - action:string Action when the condition is met, can be `continue`, `exit_success`, `exit_failed` Play the scenario only if the condition is met. For example, it can stop processing scenario if non-200 status code is encountered. #### Internal Workflow - Checks whether response status code matches the required status code codintions #### Usage ``` {"condition": {"status_code": 200}} ``` ### Wait timeout budget (ms): +${wait}Pause during the scenario to give browser some time to load. Note that the pause time is part of the scenario budget #### Parameters There are no parameters - pass the wait time in milliseconds directly #### Usage ``` {"wait": 2000} ``` ### Scroll element:string=body selector:string=bottom timeout budget (ms): +500 Scroll to the selector (or the bottom of the page if no selector provided). If the element parameter is a valid selector, the scrolling wil be executed within the selected element. The scroll is not simulated using javascript - it's created with a real mouse input. #### Internal Workflow - Waits for the selector to be visible (if set) - Waits for the element to be visible (if set) and binds scroll to the element - Scrolls the page as a human would #### Parameters - element:string=body a valid css selector or xpath or "body" - selector:string a valid css selector or xpath or "bottom" as target to scroll - infinite:int=0 infinite scroll - number of scroll iteration - click\_selector:string a valid css selector or xpath to click on after the scroll - like a "view more" button #### Usage ``` {"scroll": {"selector": "bottom"}} ``` ``` {"scroll": {"selector": "#pricing"}} ``` ``` {"scroll": {"element": "#scrollable-list", "selector": "bottom", "infinite": 2}} ``` ### Execute timeout:int=3000 timeout budget (ms): +${timeout}Execute a javascript script and store the result if a result is returned #### Internal Workflow - The Javascript code is executed - If the javascript code returns a value - it's stored and available in API response `result.browser_data.js_scenario.steps`. Note that each "execute" step has a `result` entry. - Supports async/await function #### Parameters - script:string Script to execute. It can return a serializable value - timeout:int Timeout to wait after the script execution have started - expressed in millisecond #### Usage ``` {"execute": {"script": "document.querySelector(\"body\").style.backgroundColor = \"red\";}"} ``` ``` {"execute": {"script": "return navigator.userAgent", "timeout": "1000"} ``` ### Wait For Navigation timeout:int=1000 timeout budget (ms): +${timeout} + 1500 Time to wait to detect a navigation / changing page. The given timeout + 1500 (1.5s) is added to the scenario budget - this additional time represent the average duration of a standard page loading (with assets, xhr, etc). For example if you set a timeout of 1000, 2500 is counted. #### Parameters - timeout:int Maximum timeout to wait for a navigation - expressed in millisecond #### Usage ``` {"wait_for_navigation": {}} ``` ``` {"wait_for_navigation": {"timeout": 5000}} ``` ### Wait For Selector selector:string=body state:string=visible timeout budget (ms): +${timeout} Wait the element is visible (if state=visible) in the page or the element disappear (state=hidden). If the selector is not present in the desired state until the timeout this step failed and the scenario is aborted. The timeout is added to the scenario budget #### Parameters - selector:string=body a valid css selector or xpath or "body" - state:string=visible state of the element in the page "visible" or "hidden" - timeout:int=5000 Timeout to wait before fail - expressed in milliseconds #### Usage ``` {"wait_for_selector": {"selector": "#pricing"}} ``` ``` {"wait_for_selector": {"selector": "#loading", "state": "hidden", "timeout": 10000}} ``` ### Select selector:string value:string option\_selector:string text:string index:int timeout:int=5000 timeout budget (ms): +${timeout} Select an option from a dropdown menu. Supports both native HTML ``: selects the option matching the given value - For custom dropdowns: clicks the trigger, waits for options to appear, then clicks the matching option #### Parameters - selector:string CSS or XPath selector for the dropdown trigger element - value:string For native `` elements, or `option_selector` for custom dropdowns. When using `option_selector`, optionally add `text` or `index` to narrow the match. #### Usage **Native `