Customize Request

Introduction

Scrapfly API give you the possibility to customize every aspect of scrape request. You should be able to fulfill every requirements to get the expected result.

Method

Available methods:

  • GET
  • PUT
  • POST
  • PATCH
  • HEAD

Simply call the API with the desired method and if body if needed, they will be fowarded to upstream website.

Interactive Example: HEAD Scrape Request Example
openapi openapi HEAD Scrape Request Example Sign in 
            curl -X HEAD https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything

key
=
url
= https%3A%2F%2Fhttpbin.org%2Fanything
Interactive Example: POST Scrape Request with form data
openapi openapi POST Scrape Request with form data Sign in 
            curl -X POST https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fx-www-form-urlencoded&correlation_id=fa2b4f1c-e293-4770-a03d-4a8d67760d0b --data-raw "test=1"
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fx-www-form-urlencoded&correlation_id=fa2b4f1c-e293-4770-a03d-4a8d67760d0b

key
=
url
= https%3A%2F%2Fhttpbin.org%2Fanything
headers[content-type]
= application%2Fx-www-form-urlencoded
correlation_id
= fa2b4f1c-e293-4770-a03d-4a8d67760d0b

Headers

Scrapfly API give you the possibility to customize headers sent to upstream website in very simple way.

Interactive Example: Simulate Ajax call
openapi openapi Simulate Ajax call Sign in 
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fjson&headers%5Bx-requested-with%5D=XMLHttpRequest
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fjson&headers%5Bx-requested-with%5D=XMLHttpRequest

key
=
url
= https%3A%2F%2Fhttpbin.org%2Fanything
headers[content-type]
= application%2Fjson
headers[x-requested-with]
= XMLHttpRequest

Cookies

We create a dedicated section about cookie, because most of similar Scrape API expose cookies customization as special parameters. Cookies are headers, and should not be treated as "special".

Set-Cookie

This header should never be sent from the requester site, it's a response header sent when upstream want register a cookie with some parameters (domain appliance, expiration, security ...)

Cookie

When calling upstream website as requester, you should set Cookie header. There are different variation of Cookie header.

  • Single cookie: Cookie: test=1
  • Multiple cookie: Cookie: test=1;lang=fr;currency=USD
You can also pass multiple Cookie with single notation headers to set multiple cookies.
Interactive Example: Example with 3 predefined cookies
openapi openapi Example with 3 predefined cookies Sign in 
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcookie%5D=lang%3Dfr%3Bcurrency%3DUSD%3Btest%3D1
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcookie%5D=lang%3Dfr%3Bcurrency%3DUSD%3Btest%3D1

key
=
url
= https%3A%2F%2Fhttpbin.org%2Fanything
headers[cookie]
= lang%3Dfr%3Bcurrency%3DUSD%3Btest%3D1

Geo Targeting

Geo Targeting is available, you can set the desired country via ISO 3166-1 alpha-2 of country. Available countries are defined with your proxy pool. If country is not availble from the Public Pool you can create your own private pool with desired countries.

If you want to know more about proxies, you can checkout our dedicated section

Available countries :

  • France
  • Netherlands
  • United Kingdom
  • Germany
  • Italy
  • Spain
  • Poland
  • United States
  • Canada
  • Brazil
Interactive Example: Scrape from Netherlands
openapi openapi Scrape from Netherlands Sign in 
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=http%3A%2F%2Fip-api.com%2Fjson&country=nl
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=http%3A%2F%2Fip-api.com%2Fjson&country=nl

key
=
url
= http%3A%2F%2Fip-api.com%2Fjson
country
= nl

Integration