Customize Requests

All Scrapfly HTTP requests can be customized with custom headers, methods, cookies and other HTTP parameters. Let's take a look at the available options.

Method

The scrape requests method is equivalent to the HTTP method used to call the API. For example, calling Scrapfly through POST will forward the request as a POST request to the upstream website.
Available methods are: GET, PUT, POST, PATCH, HEAD

var options = {
  method: "GET",
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fhtml", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.dev%2Fhtml"

"api.scrapfly.io"
"/scrape"

key  = "" 
url  = "https://httpbin.dev/html" 

var options = {
  method: "POST",
  body: "{\"example\":\"value\"}"
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fpost", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.dev%2Fpost"

"api.scrapfly.io"
"/scrape"

key  = "" 
url  = "https://httpbin.dev/post" 

var options = {
  method: "POST",
  body: "{\"example\":\"value\"}"
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fpost&headers[content-type]=application%2Fjson", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.dev%2Fpost&headers[content-type]=application%2Fjson"

"api.scrapfly.io"
"/scrape"

key                    = "" 
url                    = "https://httpbin.dev/post" 
headers[content-type]  = "application/json" 

var options = {
  method: "PUT",
  body: "{\"example\":\"value\"}"
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fput&headers[content-type]=application%2Fjson", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

var options = {
  method: "PATCH",
  body: "{\"example\":\"value\"}"
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fput&headers[content-type]=application%2Fjson", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

var options = {
  method: "HEAD",
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fhead", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

Headers

Request headers sent by Scrapfly can be customized through the headers parameter. Note that the value of headers must be urlencoded to prevent any side effects. When in doubt, use Scrapfly's url encoding web tool.

Note that Scrapfly can automatically configure some headers when Anti Scraping Protection is enabled and the protection bypass is triggered.

var options = {
  method: "GET",
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fheaders&headers[foo]=bar", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.dev%2Fheaders&headers[foo]=bar"

"api.scrapfly.io"
"/scrape"

key           = "" 
url           = "https://httpbin.dev/headers" 
headers[foo]  = "bar" 

Cookies

Cookies are just HTTP headers and shouldn't be treated in a special way. While most HTTP clients and libraries have a dedicated API to manage cookies, to manage cookies with Scrapfly API simply set the appropriate headers.

Set-Cookie

This header should never be sent from the client’s site. It's a response header sent when upstream wants to register a cookie with the client.

Cookie

This header contains the cookie values held by the client. So, when scraping this should be used to include cookie data. The Cookie header contains key-to-value pairs of data separated by semicolons. For example:

• Single cookie: Cookie: test=1
• Multiple cookie: Cookie: test=1;lang=fr;currency=USD

You can also pass multiple Cookie headers each with a single value to set multiple cookies.

var options = {
  method: "GET",
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fcookies&headers[cookie]=lang%3Dfr%3Bcurrency%3DUSD%3Btest%3D1", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.dev%2Fcookies&headers[cookie]=lang%3Dfr%3Bcurrency%3DUSD%3Btest%3D1"

"api.scrapfly.io"
"/scrape"

key              = "" 
url              = "https://httpbin.dev/cookies" 
headers[cookie]  = "lang=fr;currency=USD;test=1" 

Geo Targeting

Each Scrapfly request can be sent from a specific country. This is called Geo-Targetting and is managed by Scrapfly's proxy network.

The desired country can be specified 2-letter country codes (ISO 3166-1 alpha-2). Available countries are defined on the proxy pool dashboard. If the country is not available in the Public Pool a personal private pool can be created with desired countries. Note that restricting countries also restricts the available proxy IP pool.

To specify geo targetting the country parameter can be used:

• Single country selection: country=us
• Multi country selection with random selection: country=us,ca,mx
• Multi country selection with weighted random selection (higher weights have higher probability): country=us:1,ca:5,mx:3
• Country exclusion: country=-gb

For example, to send request through the United States the country=us would be used:

var options = {
  method: "GET",
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?country=us&key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fanything", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

"https://api.scrapfly.io/scrape?country=us&key=&url=https%3A%2F%2Fhttpbin.dev%2Fanything"

"api.scrapfly.io"
"/scrape"

country  = "us" 
key      = "" 
url      = "https://httpbin.dev/anything" 

For more on proxies, see the proxy documentation page

For spoofing the latitude and longitude of web browser's location services, the geolocation parameter can be used, for example: geolocation=48.856614,2.3522219 (latitude, longitude):

var options = {
  method: "GET",
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fanything", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

"https://api.scrapfly.io/scrape?key=&url=https%3A%2F%2Fhttpbin.dev%2Fanything"

"api.scrapfly.io"
"/scrape"

key  = "" 
url  = "https://httpbin.dev/anything" 

The available country options depend on the selected proxy_pool. See this table for available options for your account:

Language

Content language can be configured through the lang parameter. By default the language is inferred from the proxy location. So, if proxy of France is used the scrape request will be configured with french language preferences.

Behind the scenes, this is done by configuring the Accept-Language HTTP header. If the website supports this header and the requested language, the content will be returned in that language.

Multiple language options can be passed as well by providing multiple comma-separated values. Country locale is also supported in {lang iso2}-{country iso2} format. Note that the order matters as the website will negotiate the content language based on this order.

For example, lang=fr,en-US,en will result in final header Accept-Language: fr-{proxy country iso2},fr;q=0.9,en-US;q=0.8,en;q=0.7

Most users prefer English regardless of the proxy location. For that, use lang=en-US,en

var options = {
  method: "GET",
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?lang=en-us%2Cen&key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fanything", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

"https://api.scrapfly.io/scrape?lang=en-us%2Cen&key=&url=https%3A%2F%2Fhttpbin.dev%2Fanything"

"api.scrapfly.io"
"/scrape"

lang  = "en-us,en" 
key   = "" 
url   = "https://httpbin.dev/anything" 

Operating System

We do not recommend using this feature unless it's absolutely necessary as it can impact scraper blocking rates.

By default, Scrapfly automatically selects the most suitable Operating System for all outgoing requests. To configure operating system explicitly the os parameter can be used.

The supported values are: win,win10,win11,mac,linux,chromeos

Because of potential conflicts, the os parameter and User-Agent header cannot be set at the same time.

For example, to set Operating System to Windows 11 the os=win11 parameter would be used:

var options = {
  method: "GET",
};

try {
  const response = await fetch("https://api.scrapfly.io/scrape?os=win11&key=__API_KEY__&url=https%3A%2F%2Fhttpbin.dev%2Fanything", options);
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  }
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

"https://api.scrapfly.io/scrape?os=win11&key=&url=https%3A%2F%2Fhttpbin.dev%2Fanything"

"api.scrapfly.io"
"/scrape"

os   = "win11" 
key  = "" 
url  = "https://httpbin.dev/anything" 

Integration

• Geo Targeting example with Python SDK
• Request customization example with Python SDK
• Request customization example with Typescript SDK