JavaScript Rendering

All features available below require JavaScript rendering activated. With render_js=true from API or by ticking JavaScript rendering in API Player
JavaScript Rendering is only available with GET method.

Introduction

TL;DR Fully managed chrome headless cluster seamlessly integrated with our API and features—session, cache, and so forth.

Scrapfly provides a complete tool suite to scrape a complex/dynamic website. With the rising of the JavaScript ecosystem, more and more websites delegate to the client-side, which means the content is modified directly in the browser. Also, the version that you get with or without JavaScript displayed can be different. Most of the time, the "regular" way to scrape does not contain what you seek.

Enabling JavaScript rendering feature comes with many built-in features: xhr records, remote script execution, collect local storage and screenshot.

Web scraping with JavaScript rendering is slower than without it. It's unpredictable most of the time; it depends on the rendering time and the loaded resources. The more a website loads and executes things, the slower the scrape call will be.

Fully Managed Browser

A headless browser is fully managed, from infrastructure to fingerprinting compliance. We ensure our browser is undetectable from the anti-bot system to get a better successful scrape ratio.

JavaScript Execution

JavaScript execution gives you the ability to interact with an upstream website as a real user does. You can play scenarios, clicking buttons, filling forms, and the like. You can also collect data and get them back. You can also adjust the rendering time to ensure the termination of your script before collecting the content.

API

To inject your own JavaScript with Scrapfly API you must base64 encode your script and pass it via js parameters. You can also control the time to wait to ensure rendering is complete via rendering_wait, expressed in millisecond (Multiply second by 1000), e.g : 5000 to wait 5s. The maximum time to wait for rendering is 15s (15 000)

Example : Retrieve article title from Hacker News via Injected JavaScript

openapi openapi Injected Script | Javascript
            return Array.from(document.querySelectorAll('td.title > a')).map((el) => el.textContent)
        
Interactive Example: Javascript Execution
openapi openapi Javascript Execution Sign in
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fnews.ycombinator.com&render_js=true&js=cmV0dXJuIEFycmF5LmZyb20oZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCgndGQudGl0bGUgPiBhJykpLm1hcCgoZWwpID0%2BIGVsLnRleHRDb250ZW50KQ%3D%3D
        
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fnews.ycombinator.com&render_js=true&js=cmV0dXJuIEFycmF5LmZyb20oZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCgndGQudGl0bGUgPiBhJykpLm1hcCgoZWwpID0%2BIGVsLnRleHRDb250ZW50KQ%3D%3D

key
=
url
= https%3A%2F%2Fnews.ycombinator.com
render_js
= true
js
= cmV0dXJuIEFycmF5LmZyb20oZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCgndGQudGl0bGUgPiBhJykpLm1hcCgoZWwpID0%2BIGVsLnRleHRDb250ZW50KQ%3D%3D
openapi openapi Hacker News Extracted titles | Json
            [
    "US Travel firm $4.5m ransom negotiation open chat",
    "Laws of UX",
    "Briar Project",
    "Pleroma: A Mastodon-compatible open and federated social networking server",
    "Mastodon 3.2",
    "A philosophical difference between Haskell and Lisp",
    "Show HN: High performance X11 animated wallpapers",
    "When I raised my B2B SaaS\u2019s prices",
    "Illustrated Self-Guided Course On How To Use The Slide Rule",
    "Facebook hate-speech boycott had little effect on revenue",
    "SpaceX Crew Dragon Splashes Down in the Gulf of Mexico",
    "Why Can't We All Just Get Along? Uncertain Biological Basis of Morality (2013)",
    "\u201cZombie cicadas\u201d infected with mind-controlling fungus return to West Virginia",
    "What is a Product Roadmap?",
    "Brain-Gut Circuit Lets Microbiota Directly Affect the Sympathetic Nervous System",
    "How to Run Turing Machines on Encrypted Data [pdf]",
    "A collection of books, talks, and papers on security engineering",
    "Rethinking the Science of Skin",
    "GITenberg is an open source community for publishing ebooks in the public domain",
    "How real are real numbers? (2004)",
    "Beyond Bitswap",
    "What I Learned About Failing from My 5 Year Indie Game Dev Project",
    "The Architecture of the Medieval Page (2018)",
    "I Still Use an Old PowerPC Mac in 2020",
    "Microsoft to continue discussions on potential TikTok purchase in the US",
    "Lord and Taylor, Oldest U.S. Department Store, Files Bankruptcy",
    "\u03bcPlot v1.1 \u2013 now with log scales support",
    "OCaml for the Skeptical: OCaml in a Nutshell (2006)",
    "GPU Accelerated JavaScript",
    "Show HN: Create beautiful landing pages by copy-paste",
    "More"
]

        

Persistence

When you are using session feature, local storage and session storage data are persisted. You can also retrieve the content through the API

XHR requests

Resources such as css, jpg, png, video, etc are not tracked.

You can retrieve an emitted XHR call with associate URL, headers, body, and method. We do not attach the response. If you need the response content, you can simply directly call the XHR URL.

openapi openapi Browser Data Result | Json
            ...
"result": {
    ...,
    "browser_data": {
        "xhr_call": [
            {
                "url": "https://aan.amazon.fr/cem",
                "headers": {
                    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36",
                    "content-type": "application/json",
                    "accept": "*/*",
                    "referer": "https://images-eu.ssl-images-amazon.com/images/G/08/ape/sf/whitelisted/desktop/sf-1.50.628cb61._V408130105_.html"
                },
                "method": "POST",
                "body": "{\"render_id\":\"4a7152f0-cb58-4de8-b152-f0cb58cde8a2\",\"event_type\":\"impression\",\"dimensions\":{\"subtype\":\"impression\",\"value\":1,\"template_name\":\"Dynamic eCommerce - universal\"}}"
            },
            {
                "url": "https://www.amazon.fr/gp/customer-reviews/aj/private/reviewsGallery/get-image-gallery-assets",
                "headers": {
                    "rtt": "0",
                    "accept": "text/html,*/*",
                    "x-requested-with": "XMLHttpRequest",
                    "downlink": "10",
                    "ect": "4g",
                    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36",
                    "content-type": "application/x-www-form-urlencoded",
                    "referer": "https://www.amazon.fr/gp/product/B008AVQXDO?pf_rd_r=APG7NKFQ8DTBPK2TEN8R&pf_rd_p=70373c30-7461-4a24-bb1f-f3fde4f2df3a",
                    "cookie": "session-id=261-7851197-2783504; i18n-prefs=EUR; ubid-acbfr=262-5387700-5547500; session-id-time=2082754801l; x-wl-uid=145H5Y5j+m7oe7NpElaItmpA5YWGFqUy34ZvPnc+Yd8m+UIZC49+YTzyieSn/K4Kfq162NF1AbZo=; session-token=aLl1Sgktrzq+wYbYCVAKoXJA+3aIAhtP36mNtxkpZORbiSqd3ur/uaU6W1aHycEtUy4LpAJrcV2YmGqNHYb4trXCj3Wt4Vxc5W/aCaww5HctUNsijeRB2Dxp/ca1gtYdEEpTJGBprLlnrFg85RsOkfiWb9nysakwy54GjF9aOjksmN0ip3XCgDbO9uIZ7/X8lgM7pTDy7tTVBJtRvK79S/k9PbfDxEjXULIpNE8iYBdTvm95Xevgmgr1nouA1frzwUFYYzhCg1k=; csm-hit=tb:s-5B0K136YR4QK89MQ8RG0|1596420691120&t:1596420692684&adb:adblk_no"
                },
                "method": "POST",
                "body": null
            },
            {
                "url": "https://www.amazon.fr/gp/customer-reviews/aj/private/reviewsGallery/get-application-resources-for-reviews-gallery",
                "headers": {
                    "rtt": "0",
                    "accept": "*/*",
                    "x-requested-with": "XMLHttpRequest",
                    "downlink": "10",
                    "ect": "4g",
                    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36",
                    "content-type": "application/x-www-form-urlencoded",
                    "referer": "https://www.amazon.fr/gp/product/B008AVQXDO?pf_rd_r=APG7NKFQ8DTBPK2TEN8R&pf_rd_p=70373c30-7461-4a24-bb1f-f3fde4f2df3a",
                    "cookie": "session-id=261-7851197-2783504; i18n-prefs=EUR; ubid-acbfr=262-5387700-5547500; session-id-time=2082754801l; x-wl-uid=145H5Y5j+m7oe7NpElaItmpA5YWGFqUy34ZvPnc+Yd8m+UIZC49+YTzyieSn/K4Kfq162NF1AbZo=; session-token=aLl1Sgktrzq+wYbYCVAKoXJA+3aIAhtP36mNtxkpZORbiSqd3ur/uaU6W1aHycEtUy4LpAJrcV2YmGqNHYb4trXCj3Wt4Vxc5W/aCaww5HctUNsijeRB2Dxp/ca1gtYdEEpTJGBprLlnrFg85RsOkfiWb9nysakwy54GjF9aOjksmN0ip3XCgDbO9uIZ7/X8lgM7pTDy7tTVBJtRvK79S/k9PbfDxEjXULIpNE8iYBdTvm95Xevgmgr1nouA1frzwUFYYzhCg1k=; csm-hit=tb:s-5B0K136YR4QK89MQ8RG0|1596420691120&t:1596420692684&adb:adblk_no"
                },
                "method": "POST",
                "body": "noCache=1596420693002"
            },
            {
                "url": "https://www.amazon.fr/gp/cerberus/gv",
                "headers": {
                    "rtt": "0",
                    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36",
                    "content-type": "application/x-www-form-urlencoded",
                    "accept": "*/*",
                    "cache-control": "no-cache",
                    "x-requested-with": "XMLHttpRequest",
                    "downlink": "10",
                    "ect": "4g",
                    "referer": "https://www.amazon.fr/gp/product/B008AVQXDO?pf_rd_r=APG7NKFQ8DTBPK2TEN8R&pf_rd_p=70373c30-7461-4a24-bb1f-f3fde4f2df3a",
                    "cookie": "session-id=261-7851197-2783504; i18n-prefs=EUR; ubid-acbfr=262-5387700-5547500; session-id-time=2082754801l; x-wl-uid=145H5Y5j+m7oe7NpElaItmpA5YWGFqUy34ZvPnc+Yd8m+UIZC49+YTzyieSn/K4Kfq162NF1AbZo=; session-token=aLl1Sgktrzq+wYbYCVAKoXJA+3aIAhtP36mNtxkpZORbiSqd3ur/uaU6W1aHycEtUy4LpAJrcV2YmGqNHYb4trXCj3Wt4Vxc5W/aCaww5HctUNsijeRB2Dxp/ca1gtYdEEpTJGBprLlnrFg85RsOkfiWb9nysakwy54GjF9aOjksmN0ip3XCgDbO9uIZ7/X8lgM7pTDy7tTVBJtRvK79S/k9PbfDxEjXULIpNE8iYBdTvm95Xevgmgr1nouA1frzwUFYYzhCg1k=; csm-hit=tb:s-5B0K136YR4QK89MQ8RG0|1596420691120&t:1596420692684&adb:adblk_no"
                },
                "method": "POST",
                "body": "payload=%7B%22producerId%22%3A%22detail-page%22%2C%22asin%22%3A%22B008AVQXDO%22%2C%22asin_price%22%3A%229.49%22%2C%22asin_shipping_price%22%3A%220%22%2C%22asin_currency_code%22%3A%22EUR%22%2C%22device_type%22%3A%22WEB%22%2C%22display_code%22%3A%22Asin+is+not+eligible+because+it+has+a+retail+offer%22%2C%22substitute_count%22%3A%22-1%22%7D"
            }
        ],
        "local_storage_data": {
            "csm-hit": "tb:s-5B0K136YR4QK89MQ8RG0|1596420691120&t:1596420692684&adb:adblk_no",
            "csm:adb": "adblk_no",
            "csm-bf": "[\"5B0K136YR4QK89MQ8RG0\"]",
            "a-font-class": "a-ember"
        },
        "session_storage_data": {
            "csm-hit": "tb:s-5B0K136YR4QK89MQ8RG0|1596420691120&t:1596420692684&adb:adblk_no",
            "csm:adb": "adblk_no",
            "csm-bf": "[\"5B0K136YR4QK89MQ8RG0\"]",
            "a-font-class": "a-ember"
        },
        "javascript_evaluation_result": null
    },
    ...
}
...

        

Pricing

Using JavaScript rendering will cost 10 Scrape API calls against your quota. Keep in mind JavaScript Rendering is slow and uses many data/resources. For maximum performance, you should avoid it when it's not required.

API Response contains header X-Scrapfly-Api-Cost indicate you the billed amount