Anti Scraping Protection (ASP)

Service interruptions may occasionally occur independent of Scrapfly's control. As anti scraping protection technology is constantly evolving Scrapfly engineers are working hard to keep up with the latest changes. This may take hours to days to weeks to implement as a reliable production-grade remedy. It is essential to bear this in mind and develop your software accordingly when utilizing this feature.

Please note that we do not offer estimated time of arrival (ETA) or guarantees regarding workarounds.

Scrapfly's Anti-Scraping Protection is designed to unblock protected websites that are inaccessible to bots. We accomplish this by incorporating various concepts that help maintain a coherent fingerprint, making it as close to that of a real user as possible when scraping a website.

If you are interested in understanding the technical aspects of how we achieve this undetectability, we have published a series of articles on the subject available in the learning resources section below.

Scrapfly's Anti-Scraping Protection is powerful but not intended to assist with abuse like:
  • Automated Online Payment
  • Account Creation
  • Spam Posts
  • Vote Falsification
  • Credit Card Testing
  • Login Brute Force
  • Referral / Gifting systems
  • Ads fraud
  • Banks
  • Ticketing (Automated Buying System)
  • Betting, Casino, Gambling

These actions are prohibited by Scrapfly terms of service and will result in an immediate, non-refundable suspension of your account!

That being said, the use of ASP can be authorized for use by cybersecurity firms (red teams) after obtaining approval from the relevant parties for the specific domains they wish to test.

Scrapfly is capable of identifying and resolving obstacles posed by commonly used anti-scraping measures. Our platform also provides support for custom anti-scraping measures implemented by popular websites. Scrapfly ASP bypass does not require any extra input from you, and you will receive successful responses automatically.

It's important to keep in mind that anti-bot solutions frequently change and evolve, so our bypass techniques may need to be adjusted accordingly. It's recommended to consider ASP outages in performance-critical applications by expecting and handling relevant ASP errors correctly.

Despite the power of ASP, it might not always be enough to bypass anti-bot measures by its own when used with scrape requests that mismatch what the website is expecting. Here are some general tips for best experience:

  • When scraping POST type requests ensure that the request body and headers match what the website expects.
  • Ensure that custom website headers (usually prefixed by X-) are matched.
  • Use Residential Proxy Pools for additional bypass capabilities.
  • Enable Javascript Rendering (asp=true) for additional bypass capabilities.

If you're still experiencing issues despite all your efforts, please contact us via support chat to explore further options. Custom bypasses are available through custom plans.

Usage

When ASP is enabled, anti-bot solution vendor are automatically detected and everything is managed to bypass it.

Python HTTP 
import requests

url = "https://api.scrapfly.io/scrape?asp=true&key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fanything"
response = requests.request("GET", url)
data = response.json()
print(data)
print(data['result'])

"https://api.scrapfly.io/scrape?asp=true&key=&url=https%3A%2F%2Fhttpbin.dev%2Fanything"

"api.scrapfly.io"
"/scrape"

asp  = "true" 
key  = "" 
url  = "https://httpbin.dev/anything"

ASP bypass is magic but has limitations

While popular anti-bot vendors can be bypassed without any additional effort, there are still some areas that require manual configuration of calls.

For best results, it's important to understand how the target websites work and replicate their behavior in scraping calls. ASP bypass handles bot detection and it's up to the user to configure last mile settings to avoid identification through use patterns.

How to avoid anti bot detection on POST request

Avoiding anti-bot detection on a POST request can be tricky, but there are some key areas to focus on:

  1. Mimic a real user's behavior: Anti-bot systems often check for unusual behavior that may indicate a bot, such as a high number of requests from the same IP address or at the same time. You can mimic a real user's behavior by visiting some pages to retrieve navigation cookies/referers urls.

  2. Handling CSRF: Cross-Site Request Forgery (CSRF) is a common anti-bot measure used by websites.

    For more, see these tutorials and resources:

  3. Use realistic headers: Anti-bot systems can detect bots by looking at the headers of the requests. You should try to replicate the headers of a real user's request as closely as possible. This includes the Accept, Content-Type, Referer and Origin headers. Make sure to configure correctly the value of Accept and Content-Type regarding the content you expect (json, html).

    For more, see these tutorials and resources:

  4. Authentication: If the website requires authentication, make sure you include the correct credentials in your request. This might involve logging in to the website first, then including the session cookie or token in your POST request. If the API/Website requires it, ASP is not able to manage this, you must handle it on your side.

    For more, see these tutorials and resources:

Overall, the key to bypassing anti-bot measures on a POST request is to replicate the headers, cookies, and authentication of a regular browser request as closely as possible. This requires careful inspection of the website's code and network traffic to identify the required elements.

Website with Private/Hidden API

Scraping a private API can be a bit more challenging than scraping public APIs. Here are some recommendations to follow:

  1. Make sure you have permission: Before scraping any private API, make sure you have the necessary permission from the website owner or API provider. Scraping a private API without permission can result in legal consequences regarding the type of data exposed.
  2. Mimic a real user: When scraping a private API, it's important to mimic a real user as closely as possible. This means sending the same headers and parameters that a real user would send when accessing the API.
  3. Use authentication: Most private APIs require some form of authentication, such as a token or API key. Make sure you obtain the necessary credentials and use them in your requests.
  4. Monitor for changes: Private APIs can change over time, so it's important to monitor for any changes in the API's structure or authentication requirements. If you notice any changes, update your scraping code accordingly.

Overall, scraping private APIs requires more attention to detail and careful configuration of requests. Following these recommendations can help ensure a successful and ethical scraping process.

Maximize Your Success Rate

Network Quality

In many cases, datacenter IPs are sufficient. However, anti-bot vendors may check the origin of the IP when protecting websites, to determine if the traffic comes from a datacenter or a regular connection. In such cases, residential networks can provide a better IP reputation, as they are registered under a regular ASN that helps control the origin of the IP.

API Usage: proxy_pool=public_residential_pool, checkout the related documentation

Use a Browser

Most of anti bot check the browser fingerprint / javascript engine to generate a proof of legitimacy.

API Usage: render_js=true, checkout the related documentation

Verify Cookies and Headers

Observe headers/cookies of regular calls that are successful; you can figure out if you need to add extra headers or retrieve specific cookies to authenticate. You can use the dev tool and inspect the network activity.

API Usage: headers[referer]=https%3A%2F%2Fexample.com (value is url encoded), checkout the related documentation

To ensure navigation coherence when scraping unofficial APIs, you may need to obtain cookies from your navigation. One way to do this is by enabling session and rendering JavaScript during the initial scraping to retrieve cookies. Once the cookies are stored in your session, you can continue scraping without rendering JavaScript while still applying the previously obtained cookies for consistency. The following Scrapfly features you must take a look to achieve that:

API Usage: session=my-unique-session-name, checkout the related documentation
API Usage: render_js=true, checkout the related documentation

Geo Blocking

When browsing certain websites, users may encounter blocks based on their IP location. Scrapfly can bypass this issue by default, as it selects a random country from its pool. However, specifying the country based on the location of the website can be a helpful way to avoid geo-blocking.

API Usage: country=us, checkout the related documentation

Pricing

Our Anti-Scraping Protection (ASP) solution is a sophisticated tool that provides advanced protection against scraping attempts. It is designed to adapt to various anti-scraping measures implemented on different websites. To achieve this, the ASP dynamically mutates your configuration parameters based on the target and the anti-scraping solution in place, and this can have an impact on pricing.

The ASP can change your configuration parameters such as proxy_pool, country, session, render_js, js_scenario, rendering_wait, and os. This dynamic mutation can lead to pricing variations, particularly due to the usage of browsers and residential networks. Therefore, scraping protected websites at scale can be costly. You can check our pricing grid here

To ensure predictability and control of your spending, we recommend creating an account and gradually monitoring the usage costs as you increase your volume. Once you have determined the actual cost, you can check our set of tools to make it more predictible and ensure staying within budget.

It is essential to note that if no blocking solution is detected, there is no extra charge. Furthermore, our ASP solution is designed to lower the cost and improve response time, making it an advanced and efficient solution for your scraping needs.

Some specific protected domain have a special price due to high protection, you can ask through support or test via player to show how much credit are billed

Integration

All related errors are listed below. You can see full description and example of error response on the Errors section.

Learning Resources

Summary