Customize Request

Introduction

Scrapfly API gives you the possibility to customize every aspect of a scrape request. You should be able to fulfill every requirement to get the expected result.

Method

Available methods:

  • GET
  • PUT
  • POST
  • PATCH
  • HEAD

Simply call the API with the desired method and provide body if required / needed by method, it will be forwarded to the upstream website.

Interactive Example: HEAD Scrape Request Example
openapi openapi HEAD Scrape Request Example Sign in 
            curl -X HEAD https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything

key
=
url
= https%3A%2F%2Fhttpbin.org%2Fanything
Interactive Example: POST Scrape Request with form data
openapi openapi POST Scrape Request with form data Sign in 
            curl -X POST https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fx-www-form-urlencoded&correlation_id=fa2b4f1c-e293-4770-a03d-4a8d67760d0b --data-raw "test=1"
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fx-www-form-urlencoded&correlation_id=fa2b4f1c-e293-4770-a03d-4a8d67760d0b

key
=
url
= https%3A%2F%2Fhttpbin.org%2Fanything
headers[content-type]
= application%2Fx-www-form-urlencoded
correlation_id
= fa2b4f1c-e293-4770-a03d-4a8d67760d0b

Headers

Scrapfly API allows you to customize headers sent to the upstream website in a very simple way.

Interactive Example: Simulate Ajax call
openapi openapi Simulate Ajax call Sign in 
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fjson&headers%5Bx-requested-with%5D=XMLHttpRequest
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fjson&headers%5Bx-requested-with%5D=XMLHttpRequest

key
=
url
= https%3A%2F%2Fhttpbin.org%2Fanything
headers[content-type]
= application%2Fjson
headers[x-requested-with]
= XMLHttpRequest

Cookies

We create a dedicated section about cookies because most similar Scrape API exposes cookies customization as special parameters. Cookies are headers and should not be treated as "special."

Set-Cookie

This header should never be sent from the client’s site. It's a response header sent when upstream wants to register a cookie with some parameters (domain appliance, expiration, security, and the like.)

Cookie

When calling an upstream website as a client, you should set a Cookie header. There are different variations of Cookie header.

  • Single cookie: Cookie: test=1
  • Multiple cookie: Cookie: test=1;lang=fr;currency=USD
You can also pass multiple Cookie with a single notation headers to set multiple cookies.
Interactive Example: Example with 3 predefined cookies
openapi openapi Example with 3 predefined cookies Sign in 
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcookie%5D=lang%3Dfr%3Bcurrency%3DUSD%3Btest%3D1
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcookie%5D=lang%3Dfr%3Bcurrency%3DUSD%3Btest%3D1

key
=
url
= https%3A%2F%2Fhttpbin.org%2Fanything
headers[cookie]
= lang%3Dfr%3Bcurrency%3DUSD%3Btest%3D1

Geo Targeting

Geo-Targeting is available. You can set the desired country via ISO 3166-1 alpha-2 of the country. Available countries are defined with your proxy pool. If the country is not available from the Public Pool you can create your own private pool with desired countries.

If you want to know more about proxies, you can checkout our dedicated section

Available countries :

  • France
  • Netherlands
  • United Kingdom
  • Germany
  • Italy
  • Spain
  • Poland
  • United States
  • Canada
  • Brazil
  • Russia
  • Australia
  • Mexico
  • Japan
Interactive Example: Scrape from Netherlands
openapi openapi Scrape from Netherlands Sign in 
            curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=http%3A%2F%2Fip-api.com%2Fjson&country=nl
HTTP Call Pretty Print
https://api.scrapfly.io/scrape?key=&url=http%3A%2F%2Fip-api.com%2Fjson&country=nl

key
=
url
= http%3A%2F%2Fip-api.com%2Fjson
country
= nl

Integration