Customize Request
You can customize every aspect of a scrape request. Method, Headers (cookies), Location.
Method
Simply call the API with the desired method and provide body if required / needed by method, it will be forwarded to the upstream website.
Available methods are GET
, PUT
, POST
, PATCH
, HEAD
GET
request is the most commonly used to scrape
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything"
POST
Is commonly used to submit form. When using this method, if you do not set content-type
header, application/x-www-form-urlencoded
will be set and we assume you send urlencoded data. If you want submit json data for example, you need to specify content-type: application/json
.
curl \
--request "POST" \
--url "https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%253A%252F%252Fhttpbin.org%252Fanything&headers%5Bcontent-type%5D=application%252Fx-www-form-urlencoded&correlation_id=fa2b4f1c-e293-4770-a03d-4a8d67760d0b" \
-H content-type: text/json \
--data-raw "test=1"
"https://api.scrapfly.io/scrape?key=&url=https%253A%252F%252Fhttpbin.org%252Fanything&headers%5Bcontent-type%5D=application%252Fx-www-form-urlencoded&correlation_id=fa2b4f1c-e293-4770-a03d-4a8d67760d0b"
key = ""
url = "https://httpbin.org/anything"
headers[content-type] = "application/x-www-form-urlencoded"
correlation_id = "fa2b4f1c-e293-4770-a03d-4a8d67760d0b"
PUT
Is commonly used to submit form. When using this method, if you do not set content-type
header, application/x-www-form-urlencoded
will be set and we assume you send urlencoded data. If you want submit json data for example, you need to specify content-type: application/json
.
curl \
--request "PUT" \
--url "https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%253A%252F%252Fhttpbin.org%252Fanything" \
-H content-type: text/json \
--data-raw "test=1"
PATCH
Is commonly used to submit form. When using this method, if you do not set content-type
header, application/x-www-form-urlencoded
will be set and we assume you send urlencoded data. If you want submit json data for example, you need to specify content-type: application/json
.
curl \
--request "PATCH" \
--url "https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%253A%252F%252Fhttpbin.org%252Fanything" \
-H content-type: text/json \
--data-raw "test=1"
HEAD
request are useful to retrieve headers / status code without fetching the content. When using this method, headers of upstream website are directly forwarded
to our API response. It means the headers attached to our response, are the headers of the website you want to scrape.
curl \
--head \
--url "https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%253A%252F%252Fhttpbin.org%252Fanything"
Headers
Scrapfly API allows you to customize headers sent to the upstream website in a very simple way. The value of headers must be urlencoded to prevent side effect.
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "headers[content-type]=application/json" \
--data-urlencode "headers[x-requested-with]=XMLHttpRequest"
"https://api.scrapfly.io/scrape?key=&url=https%253A%252F%252Fhttpbin.org%252Fanything&headers%5Bcontent-type%5D=application%252Fjson&headers%5Bx-requested-with%5D=XMLHttpRequest"
key = ""
url = "https://httpbin.org/anything"
headers[content-type] = "application/json"
headers[x-requested-with] = "XMLHttpRequest"
Cookies
We create a dedicated section about cookies because most similar Scrape API exposes cookies customization as special parameters. Cookies are headers and should not be treated as "special."
Set-Cookie
This header should never be sent from the client’s site. It's a response header sent when upstream wants to register a cookie with some parameters (domain appliance, expiration, security, and the like.)
Cookie
When calling an upstream website as a client, you should set a Cookie
header. There are different variations of Cookie
header.
- Single cookie:
Cookie: test=1
- Multiple cookie:
Cookie: test=1;lang=fr;currency=USD
You can also pass multiple Cookie
with a single notation headers to set multiple cookies.
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.org/anything" \
--data-urlencode "headers[cookie]=lang=fr;currency=USD;test=1"
"https://api.scrapfly.io/scrape?key=&url=https%253A%252F%252Fhttpbin.org%252Fanything&headers%5Bcookie%5D=lang%253Dfr%253Bcurrency%253DUSD%253Btest%253D1"
key = ""
url = "https://httpbin.org/anything"
headers[cookie] = "lang=fr;currency=USD;test=1"
Geo Targeting
Geo-Targeting is available. You can set the desired country via ISO 3166-1 alpha-2 of the country. Available countries are defined with your proxy pool. If the country is not available from the Public Pool you can create your own private pool with desired countries.
If you want to know more about proxies, you can checkout our dedicated section
Available countries :
- Albania
- Armenia
- Argentina
- Austria
- Australia
- Belgium
- Bulgaria
- Bolivia
- Brazil
- Belarus
- Canada
- Switzerland
- Chile
- Colombia
- Czechia
- Germany
- Denmark
- Ecuador
- Estonia
- Spain
- Finland
- France
- United Kingdom
- Georgia
- Greece
- Croatia
- Hungary
- Ireland
- India
- Iceland
- Italy
- Japan
- South Korea
- Lithuania
- Latvia
- Mexico
- Netherlands
- Norway
- New Zealand
- Peru
- Pakistan
- Poland
- Portugal
- Romania
- Russia
- Saudi Arabia
- Sweden
- Slovakia
- Turkey
- Ukraine
- United States
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=http://ip-api.com/json" \
--data-urlencode "country=nl"
"https://api.scrapfly.io/scrape?key=&url=http%253A%252F%252Fip-api.com%252Fjson&country=nl"
key = ""
url = "http://ip-api.com/json"
country = "nl"