Web Scraping API Billing

View as markdown
Copy for LLM Open in ChatGPT Open in Claude Open in Perplexity

Simple, adaptative credit-based pricing. Each scrape request costs credits based on enabled features like JavaScript rendering, proxy type, and anti-bot protection.

How Billing Works

Credit-Based System

Each API request consumes credits based on enabled features. The total cost is returned in the X-Scrapfly-Api-Cost header and detailed in the context.cost field of the JSON response.

  • TEXT format (HTML, JSON, XML) requests are billed per request based on proxy type and features like JavaScript rendering.
  • BINARY format (images, PDFs, downloads) requests are billed based on bandwidth usage in 100KB slices.
CLOB format is billed as TEXT format requests and BLOB format is billed as BINARY format requests.

TEXT format requests are billed per request based on proxy type and enabled features (browser rendering, ASP, etc.). The result.format=TEXT field in the response confirms this billing model applies.

Feature API Credits Cost
Base Request Cost (choose one proxy pool)
Datacenter Proxies Default 1 credit
Residential Proxies proxy_pool=public_residential_pool 25 credits
Additional Features (added to base cost)
Browser Rendering render_js=true +5 credits
Note: Proxy pool costs are mutually exclusive - choose either datacenter (1 credit) or residential (25 credits) as your base cost. ASP may dynamically upgrade the proxy pool to bypass anti-bot protection, which can affect the final cost.

High Bandwidth Requests

  • Large HTML responses exceeding 1MB - Bandwidth usage after the initial 1MB is billed as BINARY bandwidth
  • Data responses (JSON, CSV, XML, TXT) exceeding 1MB - Billed as BINARY bandwidth after 1MB
  • Request body (POST/PUT/PATCH) exceeding 100KB - Sent through the API, billed as BINARY bandwidth
  • Browser rendering includes 3MB free - Additional data beyond 3MB is billed following BINARY bandwidth calculation
CDN Optimization: With browser rendering, large static files (JSON, CSS, JS) are cached on our private CDN following HTTP Cache policy. Once served from our CDN, it's not counted in bandwidth usage. Browser rendering is already optimized by preventing image loads and caching static assets.

Downloads are billed in slices of 100KB. The first megabyte is free of charge. The billed size is available in the cost details of the response context.cost field. The result.format=BINARY field in the response confirms this billing model applies.

Network Type API Credits Cost
Datacenter Proxies 3 credits per 100KB
Residential Proxies 10 credits per 100KB

Cost Calculation Example

Example: Downloading a 5MB PDF

Configuration: Datacenter proxy, 5MB file size

Calculation:
• First 1MB: Free
• Remaining 4MB = 40 slices of 100KB
• Cost: 40 × 3 = 120 credits
Total Cost: 120 credits for a 5MB PDF download

This flowchart visualizes how Web Scraping API billing works for the two distinct billing models: TEXT format (HTML, JSON, XML, etc.) and BINARY format (images, PDFs, downloads).

Each path shows the cost calculation based on your request configuration:
  • TEXT Format Path: Choose proxy pool → Add browser rendering (optional) → Apply bandwidth fees (if over 1MB)
  • BINARY Format Path: Choose proxy pool → Bandwidth-based billing per 100KB (first 1MB free)

Choose only the features you need to optimize spending

Learn More: Each scrape request features a billing section on the monitoring dashboard with a detailed breakdown of the API credits used.

Cost Examples

Here are practical examples showing how costs are calculated for different scenarios:

Basic HTML Scrape

Configuration:

Cost: 1 credit
(Datacenter proxy, no browser rendering)
With JavaScript Rendering

Configuration:

Cost: 1 + 5 = 6 credits
(Datacenter + Browser)
Residential Proxy

Configuration:

Cost: 25 credits
(Residential proxy, no browser rendering)
Full Features

Configuration:

Cost: 25 + 5 = 30 credits
(Residential + Browser)

Manage Spending

We offer a variety of tools to help you manage your spending and stay within your budget:

Project Limits

Each Scrapfly project can be restricted with:

  • Specific credit budget
  • Concurrency limits
  • Disable Pay As You Go (PAG)
Throttlers

Throttlers define limits per scraped website and timeframe:

  • Per-domain credit budget
  • Daily or monthly limits
  • Example: 10,000 credits/day for site A
Per-Call Budget

Use the cost_budget parameter:

  • Set maximum budget per request
  • Request fails if cost exceeds budget
  • Helps prevent unexpected charges
Default Pay As You Go Limit: All accounts include a hard limit for Pay As You Go to avoid major issues. PAG usage is capped at 125% of your monthly quota. You'll receive a notification when you reach 100% of your quota.

Example: With 1,000,000 API Credits/month, you can use up to 1,250,000 API Credits on PAG (total: 2,250,000 credits).

For ongoing operations reaching this limit, contact us to increase the limit exceptionally. If you're on an ENTERPRISE plan and reaching this limit, contact us to create a custom plan.

Scrape Failed Protection & Fairness Policy

Scrapfly's Scrape Failed Protection ensures that failed scrapes are not billed to our customers. To prevent abuse, we also have a fairness policy in place.

Fairness Policy Trigger: If more than 30% of failed traffic with eligible status codes (≥400 and not excluded) is detected within a minimum one-hour period, the fairness policy will be disabled and usage will be billed.
Eligible Status Codes

Status codes greater than or equal to 400 and not excluded (see below) are eligible for Scrape Failed Protection.

Excluded status codes: 400, 401, 402, 404, 405, 406, 407, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 422, 424, 426, 428, 456
Account Suspension: If an account deliberately scrapes a protected website without success and without using our Anti Scraping Protection (ASP), the account may be suspended at the discretion of our account managers.

Tracking API Credit Use

Response Headers

Each API response includes the following headers to help you track your usage:

  • X-Scrapfly-Api-Cost - API Credit cost for this request
  • X-Scrapfly-Remaining-Api-Credit - API Credits remaining in your account
  • X-Scrapfly-Project-Remaining-Api-Credit - Remaining API Credits for the project (if project has a limit)
  • X-Scrapfly-Project-Remaining-Pag-Budget - Remaining PAG Budget for the project (if project has a budget limit)
Learn More: You can also retrieve account information (quota, concurrency, etc.) via our Account API.

Cost Optimization Tips

  • Browser rendering: Only enable render_js=true if the site requires JavaScript
    Adds +5 credits per request
  • ASP (Anti-Scraping Protection): Only enable if the site has anti-bot protection
    May upgrade proxy pool automatically - see ASP documentation for details
  • Proxy pool: Use datacenter by default (1 credit), switch to residential only when needed
    Residential proxies are 25x more expensive (25 credits)

Enable caching to avoid re-scraping unchanged pages:

Savings: Cached responses cost 0 credits when hit within TTL period.

Use the cost_budget parameter to set a maximum budget per request:

  • Set the correct minimum budget for your target
  • Budget only applies to deterministic configuration (bandwidth usage cannot be known beforehand)
  • If cost budget is reached, the call is billed based on the scrape attempt settings

Set project-level limits to prevent unexpected costs:

  • Monthly credit limit - Cap total spending per month
  • Concurrency limits - Control parallel request volume
  • Disable PAG - Prevent overage charges
Learn More: See the Project documentation for details on setting up project limits.
Crawler API Billing
How crawler costs are calculated per page crawled
Extraction API Billing
AI extraction costs and options
Account Billing & Subscriptions
Payment policy, invoices, and plan management
Project Budget Management
Set spending limits and alerts
Pricing Plans
Compare plans and features

Summary