Selenium Integration

View as markdown

Selenium is the most widely-used browser automation framework supporting multiple browsers and languages. Connect it to Scrapfly Cloud Browser using the Chrome DevTools Protocol (CDP) for scalable automation with built-in proxies and fingerprinting.

Beta Feature: Cloud Browser is currently in beta.

Installation & Quick Start

Install Selenium and connect to Cloud Browser:

Installation

pip install selenium

Quick Start Example

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

API_KEY = ''
BROWSER_WS = f'wss://browser.scrapfly.io?api_key={API_KEY}&proxy_pool=datacenter&os=linux'

# Configure Chrome options for CDP connection
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", BROWSER_WS)

# Connect to Cloud Browser
driver = webdriver.Remote(
    command_executor='http://localhost:9515',  # ChromeDriver endpoint (not used for CDP)
    options=chrome_options
)

# Navigate and interact
driver.get('https://web-scraping.dev')
print('Page title:', driver.title)

# Take a screenshot
driver.save_screenshot('screenshot.png')

driver.quit()

Note: Selenium's CDP support via WebDriver BiDi is recommended for remote connections. For older Selenium versions (4.0-4.9), you may need to use alternative CDP connection methods or upgrade to Selenium 4.10+.

Installation

npm install selenium-webdriver

Quick Start Example

const { Builder } = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');

const API_KEY = '';
const BROWSER_WS = `wss://browser.scrapfly.io?api_key=${API_KEY}&proxy_pool=datacenter&os=linux`;

async function run() {
    // Configure Chrome options for CDP connection
    const options = new chrome.Options();
    options.addArguments(`--remote-debugging-address=${BROWSER_WS}`);

    // Connect to Cloud Browser
    const driver = await new Builder()
        .forBrowser('chrome')
        .setChromeOptions(options)
        .build();

    try {
        // Navigate and interact
        await driver.get('https://web-scraping.dev');
        const title = await driver.getTitle();
        console.log('Page title:', title);

        // Take a screenshot
        await driver.takeScreenshot().then(
            function(image) {
                require('fs').writeFileSync('screenshot.png', image, 'base64');
            }
        );
    } finally {
        await driver.quit();
    }
}

run();

Note: CDP connection support may vary between Selenium versions. Ensure you're using a recent version (4.10+) for best compatibility.

WebSocket Connection Parameters

The Cloud Browser WebSocket URL accepts the following query parameters:

Parameter	Required	Description	Example
`api_key`	Required	Your Scrapfly API key	`YOUR_API_KEY`
`proxy_pool`	Optional	Proxy type: `datacenter` or `residential`	`datacenter`
`os`	Optional	Operating system fingerprint: `linux`, `windows`, or `macos`	`linux`
`session`	Optional	Session ID for persistent browser state (cookies, storage)	`my-session-123`
`country`	Optional	Proxy country code (ISO 3166-1 alpha-2)	`us`, `uk`, `de`

Data Extraction

Extract data from a dynamic page using Selenium's powerful element selection:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

API_KEY = ''
BROWSER_WS = f'wss://browser.scrapfly.io?api_key={API_KEY}&proxy_pool=datacenter'

chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", BROWSER_WS)

driver = webdriver.Remote(
    command_executor='http://localhost:9515',
    options=chrome_options
)

try:
    # Navigate to the page
    driver.get('https://web-scraping.dev/products')

    # Wait for products to load
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'product'))
    )

    # Extract product data
    products = []
    product_elements = driver.find_elements(By.CLASS_NAME, 'product')

    for product in product_elements:
        title = product.find_element(By.CLASS_NAME, 'product-title').text
        price = product.find_element(By.CLASS_NAME, 'product-price').text
        url = product.find_element(By.TAG_NAME, 'a').get_attribute('href')

        products.append({
            'title': title,
            'price': price,
            'url': url
        })

    print('Products:', products)

finally:
    driver.quit()

Form Interaction

Fill forms and handle login flows using Selenium's robust element interaction:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

API_KEY = ''
BROWSER_WS = f'wss://browser.scrapfly.io?api_key={API_KEY}&proxy_pool=datacenter'

chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", BROWSER_WS)

driver = webdriver.Remote(
    command_executor='http://localhost:9515',
    options=chrome_options
)

try:
    driver.get('https://web-scraping.dev/login')

    # Wait for form to load
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, 'username'))
    )

    # Fill the login form
    username_field = driver.find_element(By.ID, 'username')
    password_field = driver.find_element(By.ID, 'password')
    submit_button = driver.find_element(By.ID, 'submit-button')

    username_field.send_keys('myuser')
    password_field.send_keys('mypassword')
    submit_button.click()

    # Wait for navigation and check if login was successful
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'user-profile'))
    )

    print('Login successful!')

finally:
    driver.quit()

Session Persistence

Maintain browser state across connections using the session parameter:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

API_KEY = ''
SESSION_ID = 'my-persistent-session'

# First connection: Login and set cookies
def first_connection():
    chrome_options = Options()
    chrome_options.add_experimental_option(
        "debuggerAddress",
        f'wss://browser.scrapfly.io?api_key={API_KEY}&session={SESSION_ID}'
    )

    driver = webdriver.Remote(
        command_executor='http://localhost:9515',
        options=chrome_options
    )

    try:
        driver.get('https://web-scraping.dev/login')
        # ... perform login ...
        print('Session created and saved')
    finally:
        driver.quit()  # Session is preserved on server


# Second connection: Reuse the logged-in session
def second_connection():
    chrome_options = Options()
    chrome_options.add_experimental_option(
        "debuggerAddress",
        f'wss://browser.scrapfly.io?api_key={API_KEY}&session={SESSION_ID}'
    )

    driver = webdriver.Remote(
        command_executor='http://localhost:9515',
        options=chrome_options
    )

    try:
        driver.get('https://web-scraping.dev/dashboard')
        print('Already logged in from previous session!')
        # Cookies and storage from first connection are preserved
    finally:
        driver.quit()


# Run both connections
first_connection()
second_connection()

Proxy Options

Proxy Pool	Use Case	Cost
`datacenter`	General scraping, high speed, lower cost	1 credits/30s + 2 credits/MB
`residential`	Protected sites, geo-targeting, anti-bot bypass	1 credits/30s + 10 credits/MB

Best Practices

Use explicit waits - Leverage WebDriverWait and expected_conditions for reliable element interactions
Handle disconnects - Wrap connections in try/except (Python) or try/catch (JavaScript)
Close browsers - Always call driver.quit() to stop billing and release resources
Use sessions wisely - Reuse sessions for multi-step flows to maintain login state
Leverage Selenium features - Use element locators (By.ID, By.CLASS_NAME, etc.) and built-in waits
Upgrade to Selenium 4.10+ - For better CDP and WebDriver BiDi support

Troubleshooting

Cause: CDP connection via debuggerAddress may not be supported in older Selenium versions.

Solution: Upgrade to Selenium 4.10+ or use WebDriver BiDi connection method.

pip install --upgrade selenium

Cause: WebSocket connection to Cloud Browser failed or timed out.

Solution: Verify your API key and check network connectivity. Ensure your firewall allows WebSocket connections.

# Test WebSocket connectivity
curl -i -N \
  -H "Connection: Upgrade" \
  -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" \
  -H "Sec-WebSocket-Key: test" \
  "https://browser.scrapfly.io?api_key="

Cause: Session ID not provided or session expired.

Solution: Always include the same session parameter in the WebSocket URL for persistent sessions.

SESSION_ID = "my-persistent-session"
BROWSER_WS = f"wss://browser.scrapfly.io?api_key={API_KEY}&session={SESSION_ID}"

Note: Sessions expire after 1 hour of inactivity by default.

Selenium Integration

Installation & Quick Start

Installation

Quick Start Example

Installation

Quick Start Example

WebSocket Connection Parameters

Data Extraction

Form Interaction

Session Persistence

Proxy Options

Best Practices

Troubleshooting

Error: "WebDriverException: invalid argument"

Connection Timeout

Session Not Persisting

Related

Summary