[Blog](https://scrapfly.io/blog) / [Web Scraping with Grok: Build an Autonomous Scraping Agent](https://scrapfly.io/blog/posts/web-scraping-with-grok) # Web Scraping with Grok: Build an Autonomous Scraping Agent by [Ziad Shamndy](https://scrapfly.io/blog/author/ziad) Jul 07, 2026 15 min read [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-grok "Share on LinkedIn") Writing CSS selectors for every new target site takes time. When a site updates its layout, every selector breaks overnight. At scale, web scraping means wrestling with brittle paths, rotating proxies, and dozens of page structures at once. Grok is xAI's large language model with native tool calling support. Grok can decide which tools to invoke, execute them in sequence, and produce structured output without requiring explicit control flow from the developer. In this guide, we build a working autonomous scraping agent using Grok and Python. We cover the xAI API setup, tool definition, the agent loop, multi-page extraction, and Scrapfly integration for sites that block plain requests. ## Key Takeaways Build an autonomous web scraping agent with Grok that fetches pages, extracts structured data, and handles pagination without writing a single explicit scraping loop. - Grok uses the xAI API, which is OpenAI-SDK-compatible, making Python setup straightforward for teams already familiar with the OpenAI client - Tool calling lets Grok decide when to fetch a page, when to extract data, and when to stop without any orchestration code in the caller - The agent loop continues until Grok returns a final answer with no pending tool calls, making it naturally self-terminating - A dedicated pagination tool lets the agent crawl multi-page listings without hardcoded loop counters or page number logic - Replacing the plain HTTP fetch tool with Scrapfly unlocks anti-bot bypass and JavaScript rendering for protected sites **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## What Is Grok? [Grok](https://x.ai/) is the large language model family developed by xAI. Grok models support long context windows, structured reasoning, and function calling, which is the capability that makes Grok suitable for building autonomous agents. ### The xAI API The [xAI API](https://docs.x.ai/) uses the same request format as the OpenAI API. Python developers can point the official [OpenAI Python SDK](https://github.com/openai/openai-python) at the xAI base URL and use the same patterns they already know. The current flagship model is `grok-3`, which handles multi-step tool calling reliably across agentic workflows. ### How Grok Tool Calling Works Tool calling gives Grok the ability to request external function execution during a conversation. The developer defines tools as JSON schemas, and Grok reads those schemas to decide which tool to call and what arguments to pass. This pattern is what makes Grok well-suited for scraping agents. Grok handles the decision layer. The Python runtime handles execution. Neither layer needs to know the internal details of the other. [Guide to Understanding and Developing LLM AgentsExplore how LLM agents transform AI, from text generators into dynamic decision-makers with tools like LangChain for automation, analysis & more!](https://scrapfly.io/blog/posts/practical-guide-to-llm-agents) With the concept clear, the next section installs the required packages and connects the client. ## Setting Up the Environment This guide requires Python 3.10 or newer. The two main packages are `openai` for the xAI client and `httpx` with `beautifulsoup4` for the scraping tools Grok will call. ### Installing Dependencies Create a virtual environment and install the required packages: bash```bash python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install openai httpx beautifulsoup4 ``` A virtual environment prevents version conflicts when these packages are updated later. ### Configuring the xAI Client The xAI API is OpenAI-compatible. The only required change from a standard OpenAI setup is the `base_url` parameter: python```python from openai import OpenAI client = OpenAI( api_key="YOUR_XAI_API_KEY", base_url="https://api.x.ai/v1" ) ``` Replace `YOUR_XAI_API_KEY` with a key from the [xAI console](https://console.x.ai/). The `client` object is the only interface the agent loop needs to communicate with Grok. Before building the agent, run a quick test to confirm the connection works: python```python response = client.chat.completions.create( model="grok-3", messages=[{"role": "user", "content": "Reply with OK if you can read this."}] ) print(response.choices[0].message.content) ``` A successful response prints `OK` and confirms the API key, base URL, and model name are all correct. With the client configured, the next step is defining the tools Grok can call during a scraping task. ## Defining Scraping Tools for Grok Each tool has two parts: a JSON schema that describes the tool to Grok, and a Python function that runs when Grok calls the tool. The schemas go into the `tools` list passed to the API. The Python functions live in the runtime and execute locally whenever Grok requests them. ### The Fetch Tool The fetch tool retrieves the raw HTML of a page. Grok calls this tool whenever the agent needs to read a URL: python```python import httpx def fetch_page(url: str) -> str: headers = { "User-Agent": ( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " "AppleWebKit/537.36 (KHTML, like Gecko) " "Chrome/124.0.0.0 Safari/537.36" ) } response = httpx.get(url, headers=headers, follow_redirects=True, timeout=15) response.raise_for_status() return response.text ``` The function sets a realistic `User-Agent` header to pass basic bot detection checks. The returned text is raw HTML, which Grok passes to the extraction tool next. ### The Extract Tool The extract tool parses product data from HTML and returns a JSON string. Grok calls this tool after fetching a page to turn raw markup into structured records: python```python import json from bs4 import BeautifulSoup def extract_products(html: str) -> str: soup = BeautifulSoup(html, "html.parser") products = [] for card in soup.select(".product"): name_el = card.select_one("h3 a") price_el = card.select_one(".price") if name_el and price_el: products.append({ "name": name_el.get_text(strip=True), "price": price_el.get_text(strip=True) }) return json.dumps(products, indent=2) ``` [BeautifulSoup](https://beautiful-soup-4.readthedocs.io/en/latest/) selects `.product-card` elements from the product listing and pulls the name and price from each card. The output is a JSON string that Grok can read and reason over in the next turn. ### The Tool Schema List Grok reads tool capabilities from a JSON schema list. Here are the schemas for both tools: python```python tools = [ { "type": "function", "function": { "name": "fetch_page", "description": "Fetch the HTML content of a webpage given its URL.", "parameters": { "type": "object", "properties": { "url": { "type": "string", "description": "The full URL of the page to fetch." } }, "required": ["url"] } } }, { "type": "function", "function": { "name": "extract_products", "description": "Extract product names and prices from raw HTML content.", "parameters": { "type": "object", "properties": { "html": { "type": "string", "description": "The raw HTML string of the product listing page." } }, "required": ["html"] } } } ] ``` Good `description` fields matter more than they appear to. Grok uses those strings to decide which tool fits a given step. Vague descriptions cause Grok to pick the wrong tool or skip calling tools entirely. With the tools defined, the agent loop ties everything together. ## Building the Autonomous Agent Loop The agent loop sends a task to Grok, executes any tool calls Grok requests, feeds the results back into the conversation, and repeats until Grok stops calling tools and returns a final answer. ### How the Loop Works At each turn the loop checks whether Grok's response includes tool calls. If tool calls are present, the loop executes each one and appends the result to the message history. If no tool calls are present, the loop returns Grok's final message as the answer. The message history is the agent's short-term memory. Every tool result stays in history so Grok can reference earlier results when deciding what to do next. ### Implementing the Loop python```python import json TOOL_REGISTRY = { "fetch_page": fetch_page, "extract_products": extract_products } def run_scraping_agent(task: str) -> str: messages = [ { "role": "system", "content": ( "You are a web scraping assistant. Use the provided tools to fetch " "web pages and extract structured data. Once you have all the data " "the user asked for, return it as a JSON array." ) }, {"role": "user", "content": task} ] while True: response = client.chat.completions.create( model="grok-3", messages=messages, tools=tools, tool_choice="auto" ) message = response.choices[0].message messages.append(message) if not message.tool_calls: return message.content for tool_call in message.tool_calls: func_name = tool_call.function.name func_args = json.loads(tool_call.function.arguments) result = TOOL_REGISTRY[func_name](**func_args) messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": result }) ``` `tool_choice="auto"` lets Grok decide whether to call a tool or return a final answer. The loop exits as soon as `message.tool_calls` is empty, which happens when Grok has enough data to respond to the original task. ### Running the Agent Call `run_scraping_agent` with a plain-language description of the scraping task: python```python result = run_scraping_agent( "Fetch the product listings at https://web-scraping.dev/products " "and return a JSON array with each product name and price." ) print(result) ``` Expected output: json```json [ {"name": "Red Energy Potion", "price": "$9.99"}, {"name": "Dark Blue Elixir", "price": "$14.99"}, {"name": "Green Stamina Potion", "price": "$8.49"} ] ``` Grok fetches the page, calls `extract_products` on the returned HTML, then returns the JSON list. No explicit step-by-step orchestration is needed in the caller. [How to Build a Web Scraping Agent with ClaudeLearn how to build a reliable web scraping agent with Claude. Covers Claude Code skills, the Anthropic API, autonomous agent mode, and Scrapfly CLI integration with step-by-step Python examples.](https://scrapfly.io/blog/posts/how-to-build-a-web-scraping-agent-with-claude) With the basic agent working, the next step adds a pagination tool so the agent can crawl all pages of a listing automatically. Scrapfly #### Scale your web scraping effortlessly Scrapfly handles proxies, browsers, and anti-bot bypass — so you can focus on data. [Try Free →](https://scrapfly.io/register)## Scraping Multi-Page Listings Most product catalogs span multiple pages. Instead of hardcoding a loop over page numbers, a pagination tool lets Grok detect whether a next page exists and decide when to follow it. ### Adding a Pagination Tool The pagination tool reads the current page HTML and returns the URL of the next page, or `null` if none exists: python```python from urllib.parse import urljoin def get_next_page_url(html: str, current_url: str) -> str: soup = BeautifulSoup(html, "html.parser") next_link = soup.select_one("a.next-page, a[rel='next'], .pagination .next a") if next_link and next_link.get("href"): href = next_link["href"] if href.startswith("http"): return href return urljoin(current_url, href) return "null" ``` The function checks common pagination link patterns. If a relative URL is found, `urljoin` converts it to an absolute URL before returning. Add the schema to the `tools` list and register the function: python```python tools.append({ "type": "function", "function": { "name": "get_next_page_url", "description": ( "Given the HTML of the current page and its URL, " "return the URL of the next page or 'null' if no next page exists." ), "parameters": { "type": "object", "properties": { "html": { "type": "string", "description": "HTML of the current page." }, "current_url": { "type": "string", "description": "URL of the current page." } }, "required": ["html", "current_url"] } } }) TOOL_REGISTRY["get_next_page_url"] = get_next_page_url ``` ### Running the Multi-Page Agent With the pagination tool registered, update the task description to tell Grok to follow pages: python```python result = run_scraping_agent( "Scrape all product listings from https://web-scraping.dev/products. " "Follow pagination to collect products from every available page. " "Return a single JSON array with all product names and prices." ) print(result) ``` Grok will fetch the first page, extract products, call `get_next_page_url` to find the next URL, fetch that page, and repeat until `get_next_page_url` returns `"null"`. The final answer is a complete list across all pages. [How to Crawl the Web with PythonIntroduction to web crawling with Python. What is web crawling? How it differs from web scraping? And a deep dive into code, building our own crawler and an example project crawling Shopify-powered websites.](https://scrapfly.io/blog/posts/crawling-with-python) The agent handles pagination on its own, but many real sites block plain HTTP requests before the agent even reaches the product data. The next section covers how Scrapfly solves that. ## Integrating Scrapfly for Anti-Bot Protection The `fetch_page` function works on sites that allow plain HTTP requests. Many sites return a Cloudflare challenge, a CAPTCHA wall, or an empty response when they detect scraper traffic patterns. [How to Bypass Anti-Bot Protection When Web ScrapingLearn how anti-bot systems detect scrapers and 5 universal bypass techniques including proxy rotation, fingerprinting, and fortified headless browsers.](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping) ### Why Plain HTTP Requests Fail Anti-bot systems check for missing browser headers, TLS fingerprints that do not match real browsers, and IP ranges associated with data centers. A plain `httpx.get` call fails most of these checks before the page even loads. Replacing the fetch function is the only change needed. The agent loop, the tool schemas, and every Grok call stay exactly the same. ### Installing the Scrapfly SDK bash```bash pip install scrapfly-sdk ``` ### Replacing the Fetch Tool with Scrapfly python```python from scrapfly import ScrapflyClient, ScrapeConfig scrapfly_client = ScrapflyClient(key="YOUR_SCRAPFLY_API_KEY") def fetch_page(url: str) -> str: result = scrapfly_client.scrape(ScrapeConfig( url=url, asp=True, render_js=True )) return result.scrape_result["content"] ``` `asp=True` activates Scrapfly's anti-scraping protection bypass. `render_js=True` runs the page in a real browser so JavaScript-rendered content appears in the returned HTML. The function signature stays identical to the original, so nothing else in the agent changes. Run the same task against the updated agent: python```python result = run_scraping_agent( "Fetch the product listings at https://web-scraping.dev/products " "and return a JSON array with each product name and price." ) print(result) ``` The Scrapfly SDK handles proxy rotation, browser fingerprinting, and CAPTCHA solving transparently. Grok does not need to know the underlying fetch mechanism changed. ## Power Up with Scrapfly Building autonomous agents with Grok and a plain HTTP library covers simple targets well. Real sites add TLS fingerprinting, behavioral analysis, IP-based blocking, and JavaScript-heavy rendering that bare HTTP cannot handle. ScrapFly's [Web Scraping API](https://scrapfly.io/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **99.99% success rate** across **130M+ proxies in 120+ countries**. - [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including Make, n8n, Zapier, LangChain, and LlamaIndex. ## FAQ What is the difference between Grok and other LLMs for scraping agents?Grok uses the same tool calling interface as OpenAI models. Agent code written for GPT-4 can switch to Grok by changing the base URL and model name. The practical differences show up in reasoning quality, rate limits, and context window size for specific workloads. Can the Grok agent handle JavaScript-rendered pages?A plain HTTP fetch tool cannot execute JavaScript. To handle JavaScript-rendered pages, replace the fetch tool with a Scrapfly call that includes `render_js=True`. The agent loop, the tool schemas, and the Grok calls do not need to change. How do I prevent the agent from running in an infinite loop?Add a step counter to the `while True` loop. Increment the counter on each tool call turn and return early once the counter exceeds a safe threshold. Ten to twenty steps is a reasonable ceiling for most scraping tasks. Should I clean the HTML before passing it to Grok as a tool result?For pages with heavy boilerplate, stripping `