Crawler API Billing

View as markdown

Copy for LLM

Copy for LLM Open in ChatGPT Open in Claude Open in Perplexity

The Crawler API billing is simple: crawler cost = sum of all Web Scraping API calls made during the crawl.

How It Works

Each page crawled is billed as a Web Scraping API request based on your enabled features.

Total crawler cost = Number of pages crawled × Cost per page

For detailed pricing rules and cost breakdown, see the Web Scraping API Billing documentation.

Cost Examples

Here are a few examples showing how crawler costs are calculated. Remember, each page follows the same billing rules as the Web Scraping API.

Example 1: Basic Crawl (100 pages, no ASP)

{
  "url": "https://example.com",
  "max_pages": 100,
  "asp": false
}

Cost: 100 pages × base cost per page = see Web Scraping API pricing

Example 2: Crawl with ASP (100 pages)

{
  "url": "https://example.com",
  "max_pages": 100,
  "asp": true
}

Cost: 100 pages × (base cost + ASP cost) = see Web Scraping API pricing

Example 3: Crawl with Residential Proxies (100 pages)

{
  "url": "https://example.com",
  "max_pages": 100,
  "proxy_pool": "public_residential_pool"
}

Cost: 100 pages × residential proxy base cost (25 credits per page) = 2500 credits total

Cost Control

Set Budget Limits

Control costs by setting hard limits on your crawl:

• max_pages - Limit total pages crawled
• max_duration - Limit crawl duration in seconds
• max_api_credit_cost - Stop crawl when credit limit is reached

{
  "url": "https://example.com",
  "max_pages": 500,
  "max_duration": 1800,
  "max_api_credit_cost": 3000
}

Project Budget Limits

Set crawler-specific budget limits in your project settings to prevent unexpected costs:

• Monthly crawler credit limit
• Per-job credit limit
• Automatic alerts when approaching limits

Cost Optimization Tips

Since each page is billed like a Web Scraping API call, you can reduce costs by:

1. Crawl Only What You Need

• Use path filtering: include_only_paths and exclude_paths
• Set page limits: max_pages to cap total pages
• Limit depth: max_depth to focus on nearby pages
• Set budget limits: max_api_credit to stop when budget is reached

2. Use Caching

Enable caching to avoid re-scraping unchanged pages:

{
  "url": "https://example.com",
  "cache": true,
  "cache_ttl": 86400
}

Cached pages cost 0 credits when hit within TTL period.

3. Choose the Right Features

• Browser rendering: Only enable render_js=true if the site requires JavaScript (adds +5 credits)
• ASP: Only enable if the site has anti-bot protection (may upgrade proxy pool)
• Proxy pool: Use datacenter by default (1 credit), switch to residential only when needed (25 credits total - 25x more expensive)
• Content extraction: Use AI-powered extraction sparingly (Template: +1, Prompt/Model: +5 credits each)

For detailed cost optimization strategies, see: Web Scraping API Cost Optimization

Billing FAQ

Q: Does pausing a crawler stop billing?

Yes. When you pause a crawler, no new pages are crawled and no new credits are consumed.

Q: Are duplicate URLs counted?

No. The crawler automatically deduplicates URLs. Each unique URL is only crawled once per job.

Q: How are robots.txt requests billed?

Robots.txt and sitemap.xml requests are free and do not consume credits.

Q: What happens if I exceed my budget limit?

The crawler automatically stops when max_api_credit_cost is reached. You can resume it by increasing the limit.

Q: Can I get a refund for a failed crawl?

Failed crawls (system errors) are automatically not billed. For other issues, contact support.

Related Documentation

• Web Scraping API Billing
• Account Billing & Subscriptions
• Project Budget Management
• Pricing Plans