# Scrapfly Documentation

## Table of Contents

### Dashboard

- [Intro](https://scrapfly.io/docs)
- [Project](https://scrapfly.io/docs/project)
- [Account](https://scrapfly.io/docs/account)
- [Workspace & Team](https://scrapfly.io/docs/workspace-and-team)
- [Billing](https://scrapfly.io/docs/billing)

### Products

#### MCP Server

- [Getting Started](https://scrapfly.io/docs/mcp/getting-started)
- [Tools & API Spec](https://scrapfly.io/docs/mcp/tools)
- [Authentication](https://scrapfly.io/docs/mcp/authentication)
- [Examples & Use Cases](https://scrapfly.io/docs/mcp/examples)
- [FAQ](https://scrapfly.io/docs/mcp/faq)
##### Integrations

- [Overview](https://scrapfly.io/docs/mcp/integrations)
- [Claude Desktop](https://scrapfly.io/docs/mcp/integrations/claude-desktop)
- [Claude Code](https://scrapfly.io/docs/mcp/integrations/claude-code)
- [ChatGPT](https://scrapfly.io/docs/mcp/integrations/chatgpt)
- [Cursor](https://scrapfly.io/docs/mcp/integrations/cursor)
- [Cline](https://scrapfly.io/docs/mcp/integrations/cline)
- [Windsurf](https://scrapfly.io/docs/mcp/integrations/windsurf)
- [Zed](https://scrapfly.io/docs/mcp/integrations/zed)
- [Roo Code](https://scrapfly.io/docs/mcp/integrations/roo-code)
- [VS Code](https://scrapfly.io/docs/mcp/integrations/vscode)
- [LangChain](https://scrapfly.io/docs/mcp/integrations/langchain)
- [LlamaIndex](https://scrapfly.io/docs/mcp/integrations/llamaindex)
- [CrewAI](https://scrapfly.io/docs/mcp/integrations/crewai)
- [OpenAI](https://scrapfly.io/docs/mcp/integrations/openai)
- [n8n](https://scrapfly.io/docs/mcp/integrations/n8n)
- [Make](https://scrapfly.io/docs/mcp/integrations/make)
- [Zapier](https://scrapfly.io/docs/mcp/integrations/zapier)
- [Vapi AI](https://scrapfly.io/docs/mcp/integrations/vapi)
- [Agent Builder](https://scrapfly.io/docs/mcp/integrations/agent-builder)
- [Custom Client](https://scrapfly.io/docs/mcp/integrations/custom-client)


#### Web Scraping API

- [Getting Started](https://scrapfly.io/docs/scrape-api/getting-started)
- [API Specification]()
- [Monitoring](https://scrapfly.io/docs/monitoring)
- [Customize Request](https://scrapfly.io/docs/scrape-api/custom)
- [Debug](https://scrapfly.io/docs/scrape-api/debug)
- [Unblocker (formerly ASP)](https://scrapfly.io/docs/scrape-api/unblocker)
- [Proxy](https://scrapfly.io/docs/scrape-api/proxy)
- [Proxy Mode](https://scrapfly.io/docs/scrape-api/proxy-mode)
- [Proxy Mode - Screaming Frog](https://scrapfly.io/docs/scrape-api/proxy-mode/screaming-frog)
- [Proxy Mode - Apify](https://scrapfly.io/docs/scrape-api/proxy-mode/apify)
- [(Auto) Data Extraction](https://scrapfly.io/docs/scrape-api/extraction)
- [Javascript Rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering)
- [Javascript Scenario](https://scrapfly.io/docs/scrape-api/javascript-scenario)
- [SSL](https://scrapfly.io/docs/scrape-api/ssl)
- [DNS](https://scrapfly.io/docs/scrape-api/dns)
- [Cache](https://scrapfly.io/docs/scrape-api/cache)
- [Batch (Multi-URL Scraping)](https://scrapfly.io/docs/scrape-api/batch)
- [Session](https://scrapfly.io/docs/scrape-api/session)
- [Webhook](https://scrapfly.io/docs/scrape-api/webhook)
- [Schedule](https://scrapfly.io/docs/scrape-api/schedule)
- [Screenshot](https://scrapfly.io/docs/scrape-api/screenshot)
- [Errors](https://scrapfly.io/docs/scrape-api/errors)
- [Timeout](https://scrapfly.io/docs/scrape-api/understand-timeout)
- [Throttling](https://scrapfly.io/docs/throttling)
- [Troubleshoot](https://scrapfly.io/docs/scrape-api/troubleshoot)
- [Billing](https://scrapfly.io/docs/scrape-api/billing)
- [FAQ](https://scrapfly.io/docs/scrape-api/faq)

#### Crawler API

- [Getting Started](https://scrapfly.io/docs/crawler-api/getting-started)
- [API Specification]()
- [Retrieving Results](https://scrapfly.io/docs/crawler-api/results)
- [WARC Format](https://scrapfly.io/docs/crawler-api/warc-format)
- [Data Extraction](https://scrapfly.io/docs/crawler-api/extraction-rules)
- [Search](https://scrapfly.io/docs/crawler-api/search)
- [Prompt & Extract](https://scrapfly.io/docs/crawler-api/prompt)
- [Auto Refresh](https://scrapfly.io/docs/crawler-api/refresh)
- [Webhook](https://scrapfly.io/docs/crawler-api/webhook)
- [Schedule](https://scrapfly.io/docs/crawler-api/schedule)
- [Billing](https://scrapfly.io/docs/crawler-api/billing)
- [Errors](https://scrapfly.io/docs/crawler-api/errors)
- [Troubleshoot](https://scrapfly.io/docs/crawler-api/troubleshoot)
- [FAQ](https://scrapfly.io/docs/crawler-api/faq)

#### Screenshot API

- [Getting Started](https://scrapfly.io/docs/screenshot-api/getting-started)
- [API Specification]()
- [Accessibility Testing](https://scrapfly.io/docs/screenshot-api/accessibility)
- [Webhook](https://scrapfly.io/docs/screenshot-api/webhook)
- [Schedule](https://scrapfly.io/docs/screenshot-api/schedule)
- [Billing](https://scrapfly.io/docs/screenshot-api/billing)
- [Errors](https://scrapfly.io/docs/screenshot-api/errors)

#### Extraction API

- [Getting Started](https://scrapfly.io/docs/extraction-api/getting-started)
- [API Specification]()
- [Rules Template](https://scrapfly.io/docs/extraction-api/rules-and-template)
- [LLM Extraction](https://scrapfly.io/docs/extraction-api/llm-prompt)
- [AI Auto Extraction](https://scrapfly.io/docs/extraction-api/automatic-ai)
- [Webhook](https://scrapfly.io/docs/extraction-api/webhook)
- [Billing](https://scrapfly.io/docs/extraction-api/billing)
- [Errors](https://scrapfly.io/docs/extraction-api/errors)
- [FAQ](https://scrapfly.io/docs/extraction-api/faq)

#### Data API


#### Proxy Saver

- [Getting Started](https://scrapfly.io/docs/proxy-saver/getting-started)
- [Fingerprints](https://scrapfly.io/docs/proxy-saver/fingerprints)
- [Optimizations](https://scrapfly.io/docs/proxy-saver/optimizations)
- [SSL Certificates](https://scrapfly.io/docs/proxy-saver/certificates)
- [Protocols](https://scrapfly.io/docs/proxy-saver/protocols)
- [Pacfile](https://scrapfly.io/docs/proxy-saver/pacfile)
- [Secure Credentials](https://scrapfly.io/docs/proxy-saver/security)
- [Billing](https://scrapfly.io/docs/proxy-saver/billing)

#### Cloud Browser API

- [Getting Started](https://scrapfly.io/docs/cloud-browser-api/getting-started)
- [Proxy & Geo-Targeting](https://scrapfly.io/docs/cloud-browser-api/proxy)
- [Unblock API](https://scrapfly.io/docs/cloud-browser-api/unblock)
- [Captcha Solver](https://scrapfly.io/docs/cloud-browser-api/captcha-solver)
- [File Downloads](https://scrapfly.io/docs/cloud-browser-api/file-downloads)
- [Session Resume](https://scrapfly.io/docs/cloud-browser-api/session-resume)
- [Human-in-the-Loop](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop)
- [Debug Mode](https://scrapfly.io/docs/cloud-browser-api/debug-mode)
- [Browser Extensions](https://scrapfly.io/docs/cloud-browser-api/extensions)
- [Credential Vault](https://scrapfly.io/docs/cloud-browser-api/credential-vault)
- [Native Browser MCP](https://scrapfly.io/docs/cloud-browser-api/mcp)
- [DevTools Protocol](https://scrapfly.io/docs/cloud-browser-api/cdp-reference)
##### Integrations

- [Puppeteer](https://scrapfly.io/docs/cloud-browser-api/puppeteer)
- [Playwright](https://scrapfly.io/docs/cloud-browser-api/playwright)
- [Selenium](https://scrapfly.io/docs/cloud-browser-api/selenium)
- [Vercel Agent Browser](https://scrapfly.io/docs/cloud-browser-api/agent-browser)
- [Browser Use](https://scrapfly.io/docs/cloud-browser-api/browser-use)
- [Stagehand](https://scrapfly.io/docs/cloud-browser-api/stagehand)
- [1Password](https://scrapfly.io/docs/cloud-browser-api/1password)

- [Billing](https://scrapfly.io/docs/cloud-browser-api/billing)
- [Errors](https://scrapfly.io/docs/cloud-browser-api/errors)


### Tools

- [Antibot Detector](https://scrapfly.io/docs/tools/antibot-detector)

### SDK

- [Golang](https://scrapfly.io/docs/sdk/golang)
- [Python](https://scrapfly.io/docs/sdk/python)
- [Rust](https://scrapfly.io/docs/sdk/rust)
- [TypeScript](https://scrapfly.io/docs/sdk/typescript)
- [Scrapy](https://scrapfly.io/docs/sdk/scrapy)

### Integrations

- [Getting Started](https://scrapfly.io/docs/integration/getting-started)
- [LangChain](https://scrapfly.io/docs/integration/langchain)
- [LlamaIndex](https://scrapfly.io/docs/integration/llamaindex)
- [CrewAI](https://scrapfly.io/docs/integration/crewai)
- [Zapier](https://scrapfly.io/docs/integration/zapier)
- [Make](https://scrapfly.io/docs/integration/make)
- [n8n](https://scrapfly.io/docs/integration/n8n)

### Academy

- [Overview](https://scrapfly.io/academy)
- [Web Scraping Overview](https://scrapfly.io/academy/scraping-overview)
- [Tools](https://scrapfly.io/academy/tools-overview)
- [Reverse Engineering](https://scrapfly.io/academy/reverse-engineering)
- [Static Scraping](https://scrapfly.io/academy/static-scraping)
- [HTML Parsing](https://scrapfly.io/academy/html-parsing)
- [Dynamic Scraping](https://scrapfly.io/academy/dynamic-scraping)
- [Hidden API Scraping](https://scrapfly.io/academy/hidden-api-scraping)
- [Headless Browsers](https://scrapfly.io/academy/headless-browsers)
- [Hidden Web Data](https://scrapfly.io/academy/hidden-web-data)
- [JSON Parsing](https://scrapfly.io/academy/json-parsing)
- [Data Processing](https://scrapfly.io/academy/data-processing)
- [Scaling](https://scrapfly.io/academy/scaling)
- [Walkthrough Summary](https://scrapfly.io/academy/walkthrough-summary)
- [Scraper Blocking](https://scrapfly.io/academy/scraper-blocking)
- [Proxies](https://scrapfly.io/academy/proxies)

---

# Monitoring

 Scrapfly offers a detailed real-time monitoring dashboard that logs all scrape requests and their results. This dashboard tracks all scrape request for the selected Scrapfly project and environment. It can be filtered and inspected for overall scraping performance:

  [See Your Monitoring Dashboard](https://scrapfly.io/dashboard/monitoring)

> **Logs Retention**
>
> | FREE | DISCOVERY | PRO | STARTUP | ENTERPRISE |
> |---|---|---|---|---|
> | Log retention 1 week | Log retention 1 week | Log retention 2 weeks | Log retention 3 weeks | Log retention 4 weeks |
>
>   [Screenshots](https://scrapfly.io/docs/scrape-api/screenshot?view=markdown), [Debug](https://scrapfly.io/docs/scrape-api/debug?view=markdown), [Cache](https://scrapfly.io/docs/scrape-api/cache?view=markdown) belongs to log and inherit of the same retention - As soon as the log is deleted, they are also deleted

## Filters

 Filters allow you to sample or search logs from the monitoring section to investigate or check that everything works as expected.

#### Time Frame

 Eight pre-configured time frames are available (past month, past week, past day, past three hours, past hour, past 30 minutes, past 15 minutes, past 5 minutes, past 5 minutes). You can also define an arbitrary time frame.

### Dimensions

You can filter the following values:

| Name | Value | Support Multiple | Description |
|---|---|---|---|
| url | **string** | Yes | Filter for URL, support glob operator with.  **e.g:**  `https://*.wikipedia.org/wiki/*` |
| success | **bool** `true` or `false` | No | Filter for success or failed request - includes >= 500 and network errors |
| domain | **string** | Yes | Filter for a domain, the domain (TLD+n) includes a subdomain, support glob operator with   **e.g.:**  `*.google.com` |
| root\_domain | **string** | yes | Filter for root domain (TLD+1), do not include subdomain, support glob operator with   **e.g.;** `goo*.com` |
| method | **string** | Yes | Filter for method, supported values : GET, PUT, POST, PATCH |
| status\_code | **int** | Yes | Filter for status code   **e.g:**  `401` or `502,504`. Wildcard is also supported `2**` |
| cost | **int** | 1 | Filter based of on cost spent on API Credits |
| origin | **string** | Yes | Filter for origin, supported values : API, SCHEDULER |
| retries | **int** | No | Filter for retries amount |
| duration | **int** | No | Filter for duration |
| error\_code | **string** | Yes | Filter by [error codes](https://scrapfly.io/docs/scrape-api/errors?view=markdown) e.g: `ERR::SCRAPE::SCENARIO_EXECUTION` |
| errored | **bool** `true` or `false` | No | Filter scrape having an error |

 For metrics that support multiple notations, simply separate with `,` like `status=200,204`

### Chaining Filters & Multi Values

 You can chain multiple filters with space.

```
status_code=200 host=*.wikipedia.org success=true duration>=5
```

 You can filter over multiple values, in this case, `OR` operator is applied.

```
status_code=401,500,502,503
```

## Operators

Following operators are supported :

- **`=`** Equal
- **`!`** Not equal
- **`>`** Greater than
- **`>=`** Greater or equal to
- **`<`** Lower than
- **`<=`** Lower or equal to

## API — Web Scraping

> **Enterprise Plan Required** The Monitoring API is only available on **Enterprise** or **Custom** subscriptions. Calls from other plans return `HTTP 402 Payment Required` with `{"reason": "Enterprise or Custom subscription required"}`. See [pricing](https://scrapfly.io/pricing) to upgrade.
>
>  **Time-window limit:** metrics queries are capped at **90 days** (start to end). Wider windows return `HTTP 403` with `{"reason": "Period too large"}`. Use `period` or split into multiple calls. The [log export](#api-logs-export) has no window limit.

### Monitoring for other products

 [  Web Scraping current

 origin=WEB\_SCRAPING\_API

 ](https://scrapfly.io/docs/monitoring)

 [  Screenshot

 origin=SCREENSHOT\_API

 ](https://scrapfly.io/docs/screenshot-api/monitoring)

 [  Extraction

 origin=EXTRACTION\_API

 ](https://scrapfly.io/docs/extraction-api/monitoring)

 [  Crawler

 origin=CRAWLER\_API

 ](https://scrapfly.io/docs/crawler-api/monitoring)

 [  Cloud Browser

 session-based

 ](https://scrapfly.io/docs/cloud-browser-api/monitoring)

> **Quick verification — does my key work?**Run this curl from your terminal. A valid Enterprise key returns a JSON envelope with `account_metrics`; a non-Enterprise key returns 402, an invalid key returns 401.
>
>
>
> ```
> curl -sk 'https://api.scrapfly.io/scrape/monitoring/metrics?key=&period=last24h&aggregation=account' | jq .account_metrics
> ```

With monitoring API you can query aggregates or domain specific metrics.

- All date are in UTC
- JSON or MSGPACK format are available, use the header `accept` to control it, `application/json` or `application/msgpack`
- Pass your API KEY via `key=xxx` query param or header `Authorization` `Bearer xxxx`

###  GET `/scrape/monitoring/metrics`

Retrieve metrics from the current subscription period, which include three aggregation levels: account, project and targets (top 100).

```
https://api.scrapfly.io/scrape/monitoring/metrics?key=&aggregation=account&period=last24h
```

#### Code Examples

   Python   TypeScript   Go   Rust   cURL

```
from scrapfly import ScrapflyClient, ScraperAPI

client = ScrapflyClient(key='')

stats = client.get_monitoring_metrics(
    aggregation=[ScraperAPI.MONITORING_ACCOUNT_AGGREGATION],
    period=ScraperAPI.MONITORING_PERIOD_LAST_24H,
)
print(stats['account_metrics'])
```

```
import { ScrapflyClient } from 'scrapfly-sdk';

const client = new ScrapflyClient({ key: '' });

const stats = await client.getMonitoringMetrics({
    aggregation: ['account'],
    period: 'last24h',
});
console.log(stats.account_metrics);
```

```
package main

import (
    "fmt"
    "log"

    scrapfly "github.com/scrapfly/go-scrapfly"
)

func main() {
    client, err := scrapfly.New("")
    if err != nil {
        log.Fatal(err)
    }

    stats, err := client.GetMonitoringMetrics(scrapfly.MonitoringMetricsOptions{
        Aggregation: []scrapfly.MonitoringAggregation{scrapfly.MonitoringAggregationAccount},
        Period:      scrapfly.MonitoringPeriodLast24h,
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(stats["account_metrics"])
}
```

```
use scrapfly_sdk::{Client, MonitoringAggregation, MonitoringMetricsOptions, MonitoringPeriod};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::builder().api_key("").build()?;
    let stats = client
        .get_monitoring_metrics(&MonitoringMetricsOptions {
            aggregation: Some(vec![MonitoringAggregation::Account]),
            period: Some(MonitoringPeriod::Last24h),
            ..Default::default()
        })
        .await?;
    println!("{}", stats["account_metrics"]);
    Ok(())
}
```

```
curl 'https://api.scrapfly.io/scrape/monitoring/metrics?key=&aggregation=account&period=last24h'
```

#### Parameters

| Name | Description |
|---|---|
| format | - `structured` default - `prometheus` |
| aggregation | Enable aggregations, possible values are:  - `account` default - `project` - `target`   You can combine multiple aggregations at once: `account,project,target` for example |
| period | This parameter is mutually exclusive with `start` and `end`.  - `last5m` - `last1h` - `last7d` - `last24h` default - `subscription` (*use current subscription period*) |
| start | The format should be `Y-m-d H:i:s`. It's mutually exclusive with `period`; `end` must also be set.    Examples: - `2024-01-01 00:00:00` |
| end | The format should be `Y-m-d H:i:s`. It's mutually exclusive with `period`; `start` must also be set.    Examples: - `2024-01-01 00:00:00` |
| group\_subdomain | Enable or disable subdomain grouping when `target` aggregation is requested. *When using subdomain grouping, provide the root domain as input. E.g: `web-scraping.dev` and not `www.web-scraping.dev`.*  - `true` default - `false` |

###  GET `/scrape/monitoring/metrics/target`

Retrieve metrics and timeseries for a given target domain.

```
https://api.scrapfly.io/scrape/monitoring/metrics/target?domain=httpbin.dev&key=&period=last24h
```

#### Code Examples

   Python   TypeScript   Go   Rust   cURL

```
from scrapfly import ScrapflyClient, ScraperAPI

client = ScrapflyClient(key='')

stats = client.get_monitoring_target_metrics(
    domain='httpbin.dev',
    period=ScraperAPI.MONITORING_PERIOD_LAST_24H,
)
print(stats)
```

```
import { ScrapflyClient } from 'scrapfly-sdk';

const client = new ScrapflyClient({ key: '' });

const stats = await client.getMonitoringTargetMetrics({
    domain: 'httpbin.dev',
    period: 'last24h',
});
console.log(stats);
```

```
package main

import (
    "fmt"
    "log"

    scrapfly "github.com/scrapfly/go-scrapfly"
)

func main() {
    client, err := scrapfly.New("")
    if err != nil {
        log.Fatal(err)
    }

    stats, err := client.GetMonitoringTargetMetrics(scrapfly.MonitoringTargetMetricsOptions{
        Domain: "httpbin.dev",
        Period: scrapfly.MonitoringPeriodLast24h,
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(stats)
}
```

```
use scrapfly_sdk::{Client, MonitoringPeriod, MonitoringTargetMetricsOptions};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::builder().api_key("").build()?;
    let stats = client
        .get_monitoring_target_metrics(&MonitoringTargetMetricsOptions {
            domain: "httpbin.dev".into(),
            group_subdomain: false,
            period: Some(MonitoringPeriod::Last24h),
            start: None,
            end: None,
        })
        .await?;
    println!("{}", stats);
    Ok(())
}
```

```
curl 'https://api.scrapfly.io/scrape/monitoring/metrics/target?domain=httpbin.dev&key=&period=last24h'
```

#### Parameters

| Name | Description |
|---|---|
| domain   required | - `httpbin.dev` - `web-scraping.dev` - `...` |
| period | This parameter is mutually exclusive with `start` and `end`.  - `last5m` default - `last1h` - `last7d` - `last24h` - `subscription` (*use current subscription period*) |
| group\_subdomain | Enable or disable subdomain grouping. *When using subdomain grouping, provide the root domain as input. E.g: `web-scraping.dev` and not `www.web-scraping.dev`.*  - `true` default - `false` |
| start | The format should be `Y-m-d H:i:s`. It's mutually exclusive with `period`; `end` must also be set.    Examples: - `2024-01-01 00:00:00` |
| end | The format should be `Y-m-d H:i:s`. It's mutually exclusive with `period`; `start` must also be set.    Examples: - `2024-01-01 00:00:00` |

###  GET `/scrape/monitoring/logs/export`

 Download the logs of a time window as a CSV file compressed with gzip (`logs.csv.gz`), one row per log, the most recent first. It holds the logs of the project and environment of the API key, as the **Export** button of the [monitoring dashboard](https://scrapfly.io/dashboard/monitoring) does.

- Calls from other plans than **Enterprise** or **Custom** return `HTTP 402 Payment Required`.
- A `from` or `to` that is not a date, or a filter outside the grammar below, returns `HTTP 400` with the reason in `message`.
- The window has no size limit: the 90-day cap of the metrics endpoints does not apply to the export.
- An account runs up to 5 exports at the same time: a sixth returns `HTTP 429`.

#### Code Examples

   cURL   Python

```
curl -G "https://api.scrapfly.io/scrape/monitoring/logs/export" \
    --data-urlencode "key=" \
    --data-urlencode "from=2024-01-01 00:00:00" \
    --data-urlencode "to=2024-01-01 12:00:00" \
    --data-urlencode 'filters={"status": ["status_code", "MATCH", ["403", "429"]], "pages": ["url", "ANYTHING_IN", "https://web-scraping.dev/product/*"]}' \
    -o logs.csv.gz
```

```
import csv
import gzip
import json

import requests

with requests.get(
    "https://api.scrapfly.io/scrape/monitoring/logs/export",
    params={
        "key": "",
        "from": "2024-01-01 00:00:00",
        "to": "2024-01-01 12:00:00",
        "filters": json.dumps({"status": ["status_code", "MATCH", ["403", "429"]]}),
    },
    stream=True,
) as response:
    response.raise_for_status()
    # The body is the gzip file itself, read here as it downloads.
    with gzip.open(response.raw, "rt", encoding="utf-8", newline="") as file:
        for row in csv.DictReader(file):
            print(row["event_time"], row["status_code"], row["client_ip"])
```

#### Parameters

| Name | Description |
|---|---|
| from   required | Start of the window, UTC, included. The format should be `Y-m-d H:i:s`.    Examples: - `2024-01-01 00:00:00` |
| to   required | End of the window, UTC, included. The format should be `Y-m-d H:i:s`.    Examples: - `2024-01-01 12:00:00` |
| filters | A JSON object of filters, each a `[field, operator, value]` array: a log is exported when it matches all of them. The keys only name the filters.  - `{"status": ["status_code", "MATCH", ["403", "429"]]}` - `{"pages": ["url", "ANYTHING_IN", "https://web-scraping.dev/product/*"]}` |

#### Filters

 Send every value as a JSON string, or as an array of strings for a list: numbers and booleans too, `"401"` and `"true"`.

- **`EQUAL`**, **`NOT_EQUAL`**: equal, or not, to the value.
- **`MATCH`**, **`NOT_MATCH`**: equal to one of the values of the list, or to none of them.
- **`GREATER_THAN`**, **`GREATER_THAN_OR_EQUAL`**, **`LOWER_THAN`**, **`LOWER_THAN_OR_EQUAL`**: compare a number.
- **`ANYTHING_IN`**: matches a glob, case-sensitive, where `*` and `%` stand for any characters and `_` for any one character; a list matches any of its globs. A `\` before `%`, `_` or `\` matches that character itself, written `\\` in the JSON: `"https://web-scraping.dev/product\\_*"`.
- **`ANYTHING_NOT_IN`**: does not match the glob. A list keeps a log that misses any one of its globs: to leave out several, send one filter per glob.
- **`IS`**, **`IS_NOT`**: the field has no value, or has one; the value is ignored, send `"null"`. Only `country`, `correlation_id`, `error_code` and `status_code` can be without a value: on another field `IS` matches no log and `IS_NOT` every log.

| Field | Operators | Description |
|---|---|---|
| url | text | The scraped URL |
| domain | text | The host of the URL, subdomain included |
| root\_domain | text | The domain of the URL without its subdomains |
| method | text | The HTTP method, in any case: `GET`, `POST`, ... |
| country | text | The country of the proxy, e.g. `us` |
| origin | text | What sent the scrape, e.g. `WEB_SCRAPING_API`, `SCHEDULER`, `WEBHOOK` |
| correlation\_id | text | The `correlation_id` of the scrape |
| error\_code | text | The [error code](https://scrapfly.io/docs/scrape-api/errors?view=markdown) of the scrape, e.g. `ERR::SCRAPE::SCENARIO_EXECUTION` |
| status\_code | number | The status code of the scraped page |
| retries | number | The retries of the scrape |
| cost | number | The API credits of the scrape |
| duration | number | The duration of the scrape, in seconds |
| success | `EQUAL`, `NOT_EQUAL` | `"true"` or `"false"`: whether the scrape succeeded, strictly: every failed scrape is `"false"`, also one the `succeed` column counts as a success |
| errored | `EQUAL` | `"true"` or `"false"`: whether the scrape has an error code |
| js | `EQUAL` | `"true"` or `"false"`: whether the scrape rendered the page in a browser |
| tags | `MATCH` | A list of tags: the scrape carries all of them |

 A **text** field takes `EQUAL`, `NOT_EQUAL`, `MATCH`, `NOT_MATCH`, `ANYTHING_IN`, `ANYTHING_NOT_IN`, `IS` and `IS_NOT`; a **number** field `EQUAL`, `NOT_EQUAL`, the four comparisons, `MATCH`, `NOT_MATCH`, `IS` and `IS_NOT`.

#### Columns

 The first line of the file names the columns. Fields are separated by `,`; a field holding a `,`, a double quote or a line break is written between double quotes, a double quote in it doubled (RFC 4180), which any CSV reader reads back as it was.

| Name | Description |
|---|---|
| event\_time | When the scrape was logged, UTC, `Y-m-d H:i:s` |
| ulid | The ID of the log. A scrape that timed out in the API is logged by the API, and can be logged again by the scrape engine when it ends: two rows with the same `ulid` |
| method | The HTTP method of the scrape |
| url | The scraped URL |
| country | The country of the proxy |
| duration | The duration of the scrape, in seconds |
| succeed | `true` when the scrape counts as a success in the success rate: it succeeded, or it failed on a throttle (`ERR::THROTTLE::*`), webhook or browser driver error (screenshot, page element, JavaScript, scenario), whose `error_code` it carries. `false` otherwise. The `success` filter tests the strict outcome instead |
| error\_code | The error code of the scrape, empty without an error |
| cost | The API credits of the scrape |
| proxy\_pool | The proxy pool of the scrape |
| status\_code | The status code of the scraped page, empty without a response |
| project | The project of the API key |
| env | The environment of the API key, `LIVE` or `TEST` |
| correlation\_id | The `correlation_id` of the scrape |
| tags | The tags of the scrape, separated by `,` |
| client\_ip | The IP address the scrape was requested from, read from its log. Only the first 10,000 rows of the file, the most recent, can carry it, and only while their log is [retained](#log-retention) (4 weeks at most). It is empty: - on every row after the first 10,000; - when the log is deleted at the end of its retention, or was never stored; - when no client address was recorded, as for a scheduled scrape; - when the recorded value is not an IP address; - on every row after a failed read of the logs: the rest of the file goes out without IPs. |