Customize Request
Introduction
Scrapfly API give you the possibility to customize every aspect of scrape request. You should be able to fulfill every requirements to get the expected result.
Method
Available methods:
- GET
- PUT
- POST
- PATCH
- HEAD
Simply call the API with the desired method and if body if needed, they will be fowarded to upstream website.
Interactive Example: HEAD Scrape Request Example
curl -X HEAD https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything
HTTP Call Pretty Print
Interactive Example: POST Scrape Request with form data
curl -X POST https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fx-www-form-urlencoded&correlation_id=fa2b4f1c-e293-4770-a03d-4a8d67760d0b --data-raw "test=1"
HTTP Call Pretty Print
Headers
Scrapfly API give you the possibility to customize headers sent to upstream website in very simple way.
Interactive Example: Simulate Ajax call
curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcontent-type%5D=application%2Fjson&headers%5Bx-requested-with%5D=XMLHttpRequest
HTTP Call Pretty Print
Cookies
We create a dedicated section about cookie, because most of similar Scrape API expose cookies customization as special parameters. Cookies are headers, and should not be treated as "special".
Set-Cookie
This header should never be sent from the requester site, it's a response header sent when upstream want register a cookie with some parameters (domain appliance, expiration, security ...)
Cookie
When calling upstream website as requester, you should set Cookie
header. There are different variation of Cookie
header.
- Single cookie:
Cookie: test=1
- Multiple cookie:
Cookie: test=1;lang=fr;currency=USD
You can also pass multiple Cookie
with single notation headers to set multiple cookies.
Interactive Example: Example with 3 predefined cookies
curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.org%2Fanything&headers%5Bcookie%5D=lang%3Dfr%3Bcurrency%3DUSD%3Btest%3D1
HTTP Call Pretty Print
Geo Targeting
Geo Targeting is available, you can set the desired country via ISO 3166-1 alpha-2 of country. Available countries are defined with your proxy pool. If country is not availble from the Public Pool you can create your own private pool with desired countries.
If you want to know more about proxies, you can checkout our dedicated section
Available countries :
- France
- Netherlands
- United Kingdom
- Germany
- Italy
- Spain
- Poland
- United States
- Canada
- Brazil
Interactive Example: Scrape from Netherlands
curl -X GET https://api.scrapfly.io/scrape?key=__API_KEY__&url=http%3A%2F%2Fip-api.com%2Fjson&country=nl