Javascript Rendering

Scrapfly's headless browser feature is the ultimate solution for web scraping needs that involve javascript-rendered content. Each Scrapfly scrape request that has render_js option enabled runs on a dedicated cloud browser instance that is optimized for web scraping and responding quickly and reliably to web scraping actions.

Scrapfly's advanced cache resource is powered by a global private CDN for maximum efficiency, and our solution is designed to handle proxy peering with ease.

Scraping using cloud browsers is slower and requires more proxy resources. The scraping time depends on factors such as proxy location, website hosting distance, content size, number of page resources, and javascript execution time.

When Javascript Rendering is enabled, Scrapfly also tracks web resources like:

  • Browser's HTTP queries (background requests), both request and response data
  • Local Storage
  • Session Storage
  • Screenshot (on demand)
  • Remote javascript execution result (on demand)
  • Websockets (Upgrade request and dataframes)

Javascript rendering also enables advanced Scrapfly features like Javascript Scenarios which allow issuing common browser control commands like button clicks and form inputs.

When to Use Javascript Rendering?

Many modern websites require Javascript to work and load pages through javascript powered techniques like XHR (background requests). So, the most reliable way to tell is to try scraping without javascript rendering, check whether desired content exists and compare it with javascript rendering enabled. For that, use Scrapfly's web player to experiment with various configurations real-time!

Wait For Your Content

Rendering delay is vital for scraping slowly loaded content as dynamic pages take time to fully load all the content.

For slower elements the Scrapfly browsers can be instructed to wait a bit longer through explicit delay or waiting for a specific element to appear:

In this example, Scrapfly will wait 5s before extracting the content of the page. The rendering_wait parameter is expressed in milliseconds. The maximum allowed time to wait is 25s. 

curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "render_js=true" \
--data-urlencode "rendering_wait=5000" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://web-scraping.dev/product/1"
  • CSS Selector and Xpath are case sensitive
  • Characters like ~, :, / need to be escaped with \\ Example: #selector:1234 becomes #selector\\:1234

In this example, Scrapfly will wait until product reviews load indicated by the visible presence of #reviews CSS selector. The selector watcher will timeout after 15s. Selectors are case-sensitive and need to be urlencoded. 

curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "render_js=true" \
--data-urlencode "wait_for_selector=#reviews" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://web-scraping.dev/product/1"

Alternatively, XPath selectors can be used as well which are often preferred because of more advanced querying features like selecting values based on text content. For example we can wait for reviews containing the word "delicious" to load //div[contains(@class,"review")]/p[contains(text(),"delicious")] 

curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "render_js=true" \
--data-urlencode "wait_for_selector=//div[contains(@class,\"review\")]/p[contains(text(),\"delicious\")]" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://web-scraping.dev/product/1"
Related API errors :

To wait for an intermediate request to respond, you have to prefix the selector with xhr: 

curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "tags=player,project:default" \
--data-urlencode "asp=true" \
--data-urlencode "render_js=true" \
--data-urlencode "wait_for_selector=xhr:/api/graphql" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://web-scraping.dev/reviews"

XHR pattern will match as soon it's find in the URL of the XHR requests. The pattern is case sensitive and support wildcard * character. E.g: xhr:/page/*/reviews*

Javascript Execution

Scrapfly provides a way to inject javascript code to Scrapfly browsers through the js parameter

The provided javascript code has to be base64 encoded, then it'll be executed after rendering wait and before wait_for_selector (if any).

The return value of the provided javascript code is also returned by Scrapfly API (as long as it's serializable): result.browser_data.javascript_evaluation_result

For example, this JS script used on new.ycombinator.com page will retrieve all article titles

It should be encoded as base64 (see our base64 web tool )

and finally it can be passed to Scrapfly API as demonstrated in this example:

Curl HTTP 
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "render_js=true" \
--data-urlencode "js=cmV0dXJuIEFycmF5LmZyb20oZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCgnLnJldmlldyA-IHAnKSkubWFwKChlbCkgPT4gZWwudGV4dENvbnRlbnQp" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://web-scraping.dev/product/1"
https://api.scrapfly.io/scrape?render_js=true&js=returnArrayfromdocumentquerySelectorAllreviewpmapeleltextContent&key=&url=https%253A%252F%252Fweb-scraping.dev%252Fproduct%252F1

The API returns the results under result.browser_data.javascript_evaluation_result key which contains the script's return value:

Snippets

Here are some common javascript execution snippets used in web scraping:

  • Scroll to the bottom of the page to fully render the HTML rendering on the website:

Screenshot

Javascript rendering also allows you to take a screenshot of the page using the screenshot parameter. For more see our in-depth docs for screenshot capture.

Resource Tracking

Scrapfly tracks all background requests performed by the scraped websites (XHR). The captured data includes all necessary request details like the URL, headers, method, cookies and the request body. This data is available under result.browser_data.

Scrapfly also tracks browser's Local Storage and Session Storage content which is available under local_storage_data and session_storage_data keys. See this example response:

Limitations

Javascript rendering feature is only available with GET method. You can't use a browser to send POST, PATCH, PUT, HEAD requests.

Following XHR / Fetched resources are not tracked by scrapfly browsers:

  • Fonts: .woff, .woff2, .otf, .ttf
  • Media: .webm, .oga, .aac, .m4a, .mp3, .wav, .mp4
  • Image: .svg, .png, .gif, .jpg, .jpeg, .ico
  • Style: .css
  • Other: .pbf
It's not possible to directly download the media/image content with a scrapfly browser as it will load the image url as a html document by wrapping image data as img tag. To directly download images disable the browser rendering and the image data will be returned as a base64-encoded binary data.

All related errors are listed below. You can see the full description and examples of errors response on Errors section

Pricing

Using JavaScript rendering will cost 5 Scrape API Credits. Keep in mind JavaScript Rendering is slow and uses a lot of resources so for maximum performance, you should avoid it when it's not required.

API Response contains header X-Scrapfly-Api-Cost indicate you the billed amount.

Summary