Customize Request

You can customize every aspect of a scrape request. Method, Headers (cookies), Location.

Method

Simply call the API with the desired method and provide body if required / needed by method, it will be forwarded to the upstream website.
Available methods are GET, PUT, POST, PATCH, HEAD

GET request is the most commonly used to scrape

Sign in 
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything"

POST Is commonly used to submit form. When using this method, if you do not set content-type header, application/x-www-form-urlencoded will be set and we assume you send urlencoded data. If you want submit json data for example, you need to specify content-type: application/json.

Sign in 
curl \
--request "POST" \
--url "https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fx-www-form-urlencoded&correlation_id=fa2b4f1c-e293-4770-a03d-4a8d67760d0b" \
--data-raw "test=1"
"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fx-www-form-urlencoded&correlation_id=fa2b4f1c-e293-4770-a03d-4a8d67760d0b"

key                    = "" 
url                    = "https://httpbin.org/anything" 
headers[content-type]  = "application/x-www-form-urlencoded" 
correlation_id         = "fa2b4f1c-e293-4770-a03d-4a8d67760d0b"

PUT Is commonly used to submit form. When using this method, if you do not set content-type header, application/x-www-form-urlencoded will be set and we assume you send urlencoded data. If you want submit json data for example, you need to specify content-type: application/json.

Sign in 
curl \
--request "PUT" \
--url "https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything" \
--data-raw "test=1"

PATCH Is commonly used to submit form. When using this method, if you do not set content-type header, application/x-www-form-urlencoded will be set and we assume you send urlencoded data. If you want submit json data for example, you need to specify content-type: application/json.

Sign in 
curl \
--request "PATCH" \
--url "https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything" \
--data-raw "test=1"

Headers

Scrapfly API allows you to customize headers sent to the upstream website in a very simple way. The value of headers must be urlencoded to prevent side effect.

Sign in 
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "headers[content-type]=application/json" \
--data-urlencode "headers[x-requested-with]=XMLHttpRequest"
"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fjson&headers%5Bx-requested-with%5D=XMLHttpRequest"

key                        = "" 
url                        = "https://httpbin.org/anything" 
headers[content-type]      = "application/json" 
headers[x-requested-with]  = "XMLHttpRequest"

Cookies

We create a dedicated section about cookies because most similar Scrape API exposes cookies customization as special parameters. Cookies are headers and should not be treated as "special."

Set-Cookie

This header should never be sent from the client’s site. It's a response header sent when upstream wants to register a cookie with some parameters (domain appliance, expiration, security, and the like.)

Cookie

When calling an upstream website as a client, you should set a Cookie header. There are different variations of Cookie header.

  • Single cookie: Cookie: test=1
  • Multiple cookie: Cookie: test=1;lang=fr;currency=USD
You can also pass multiple Cookie with a single notation headers to set multiple cookies.
Sign in 
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "headers[cookie]=lang=fr;currency=USD;test=1"
"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcookie%5D=lang%3Dfr%3Bcurrency%3DUSD%3Btest%3D1"

key              = "" 
url              = "https://httpbin.org/anything" 
headers[cookie]  = "lang=fr;currency=USD;test=1"

Geo Targeting

Geo-Targeting is available. You can set the desired country via ISO 3166-1 alpha-2 of the country. Available countries are defined with your proxy pool. If the country is not available from the Public Pool you can create your own private pool with desired countries.

If you want to know more about proxies, you can checkout our dedicated section

Available countries :

  • France
  • Netherlands
  • United Kingdom
  • Germany
  • Italy
  • Spain
  • Poland
  • United States
  • Canada
  • Brazil
  • Russia
  • Australia
  • Mexico
  • Japan
Sign in 
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=http://ip-api.com/json" \
--data-urlencode "country=nl"
"https://api.scrapfly.io/scrape?key=&url=http%3A%2F%2Fip-api.com%2Fjson&country=nl"

key      = "" 
url      = "http://ip-api.com/json" 
country  = "nl"

Integration