[Blog](https://scrapfly.io/blog) / [ai](https://scrapfly.io/blog/tag/ai) / [Web Scraping with Hermes Agent](https://scrapfly.io/blog/posts/web-scraping-with-hermes-agent) # Web Scraping with Hermes Agent by [Hisham Medhat](https://scrapfly.io/blog/author/hisham) Jul 07, 2026 14 min read [\#ai](https://scrapfly.io/blog/tag/ai) [\#api](https://scrapfly.io/blog/tag/api) [\#blocking](https://scrapfly.io/blog/tag/blocking) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-hermes-agent "Share on LinkedIn") **Web Scraping API**Scrape any website with anti-bot bypass, proxy rotation, and JS rendering. [ Learn More ](https://scrapfly.io/products/web-scraping-api) [ Docs ](https://scrapfly.io/docs/scrape-api/getting-started) Your Hermes Agent handles `web_search` and easy pages fine. Point it at a Cloudflare-protected page and it stalls, or returns a "verify you're human" screen instead of data. On a VPS it's worse, since many sites ban server IP ranges outright. The fix isn't a better prompt; it's a cloud fetch layer that clears those blocks before they reach your agent. This guide adds Scrapfly to Hermes through a skill or an MCP server, with a decision matrix and a working example. Let's get started. [Guide to Understanding and Developing LLM AgentsExplore how LLM agents transform AI, from text generators into dynamic decision-makers with tools like LangChain for automation, analysis & more!](https://scrapfly.io/blog/posts/practical-guide-to-llm-agents) ## Key Takeaways - Hermes ships with `web_search`, `web_extract`, browser tools, and an optional official Scrapling skill. - Defaults fail three ways: plain-HTTP fallback, anti-bot stalls, and VPS IP bans. - The fix is to separate the agent's logic from the bypass mechanics, then add a cloud layer. - Install via `hermes skills install scrapfly/skills/scraper --force`, or an MCP server. - Route by job: Firecrawl for easy, Scrapling for local, Scrapfly for protected sites. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## What Web Scraping Tools Does Hermes Agent Ship With? Hermes Agent ships with two web tools, an interactive browser toolset, and one official optional skill from Nous Research. The web backend is Firecrawl by default, and the browser layer is configurable across four providers. The two built-in web tools are `web_search` and `web_extract`. `web_search` queries the web through Firecrawl. `web_extract` pulls page content as clean Markdown. With Firecrawl configured, `web_extract` gets JavaScript rendering and some anti-bot coverage. When Firecrawl isn't configured, its own docs are blunt: "Without Firecrawl configured, web\_extract falls back to plain HTTP and extracts content from raw HTML." That fallback is where JavaScript-heavy pages turn into empty shells. The browser toolset includes `browser_navigate`, `browser_snapshot`, `browser_vision`, and others (click, type, scroll, CDP passthrough) for interactive automation. You pick the provider in [the browser config](https://hermes-agent.nousresearch.com/docs/user-guide/features/browser). The choices are `firecrawl` (default), `browserbase` (managed anti-bot cloud), `browser_use` (cloud), and `camofox` (a local Firefox fork with fingerprint spoofing). On top of that, Nous Research publishes an [official Scrapling skill](https://hermes-agent.nousresearch.com/docs/user-guide/skills/optional/research/research-scrapling). You install it with `hermes skills install official/research/scrapling`. The skill runs locally in Python and adds StealthyFetcher for Cloudflare Turnstile, a Playwright-backed JavaScript fetcher, and a spider framework. For the full tool list, see [the Hermes tools reference](https://hermes-agent.nousresearch.com/docs/user-guide/features/tools). So out of the box you get cloud fetching through Firecrawl, local stealth through Camofox, and Python stealth through Scrapling. Three paths cover a lot. They don't cover everything. ## Why the Default Stack Breaks on Protected Sites The default stack breaks in three predictable ways: - `web_extract` drops to plain HTTP and loses rendered content. - Anti-bot challenges stall browser runs until they time out. - VPS-hosted agents hit IP-range bans that no stealth setting fixes. These three failure modes show up the moment you target a serious site. The first failure mode is the plain-HTTP fallback. When Firecrawl isn't configured or gets rate-limited, `web_extract` fetches raw HTML and the page's JavaScript never runs. You asked for the article body and got a blank container. The second failure mode is anti-bot challenges. As one r/hermesagent user, CapMonster1, put it in a [thread about ditching Hermes over scraping failures](https://www.reddit.com/r/hermesagent/comments/1tjjoog/multiple_scraping_issues_multiple_solutions_tried/): "This is basically modern scraping in a nutshell: every workaround fixes one problem and creates two new ones. Firecrawl may help with captchas, but the WAF can still return empty content, and browser automation gets detected way more aggressively now than it did a few years ago." The same wall appears whether you route through [Firecrawl](https://scrapfly.io/compare/firecrawl-alternative), [Browserbase](https://scrapfly.io/compare/browserbase-alternative), Browser Use, Camofox, or Scrapling. Each one hits a Turnstile, DataDome, or hCaptcha challenge eventually. The third failure mode is VPS IP bans. Hermes users on Hostinger, AWS, or DigitalOcean run into this on day one. Many sites ban whole VPS IP ranges outright, and no stealth setting fixes a flagged IP. (The r/hermesagent HELP and Infra/Hosting flairs are full of these reports.) Stealth doesn't fix IP reputation, because the IP itself is the signal. All three failure modes matter together, because price monitoring, news aggregation, and public-record collection all touch sites that fall into one or more buckets. The fix the thread keeps circling back to is to separate the agent's logic from the bypass mechanics. For the deeper version of why challenges fire, see our [Cloudflare bypass guide](https://scrapfly.io/blog/posts/how-to-bypass-cloudflare-anti-scraping). ## How Scrapfly Fits Into the Hermes Stack Scrapfly is the cloud fetch layer your Hermes agent calls when the defaults run out of room. It isn't a Firecrawl replacement and it isn't a Scrapling replacement. It's the escalation path for the three failure modes above. The design rule is simple. Separate the agent's logic from the bypass mechanics. Hermes handles planning, prompting, memory, and skills. Scrapfly handles the fetch, including anti-bot bypass, JavaScript rendering, proxy rotation, and challenge solving on the server side. In Scrapfly terms, that split buys you four things: - Managed residential proxies: 130M+ proxies across 190+ countries, so your VPS IP never touches the target. - A 98% success rate against Cloudflare Bot Management. - Pay-only-on-success billing, where failed requests are free. - One credit pool across five skills: scraper, extraction, screenshot, crawler, and browser. For a typical Hermes setup, you keep Firecrawl for simple `web_search` and lightweight `web_extract`, and keep Scrapling for local Python stealth. You add Scrapfly for protected sites and for jobs the built-ins don't cover, like screenshots, extraction, and full crawls. Many users keep all three and let skill rules route per task. The [Web Scraping API](https://scrapfly.io/web-scraping-api) powers the Scrapfly fetch. ## Install the Scrapfly Skill in Hermes Agent Install a Scrapfly skill in your Hermes workspace with `hermes skills install`, or wire up the Scrapfly MCP server in `~/.hermes/config.yaml`. Both paths give your agent direct access to Scrapfly's scrape, extract, screenshot, crawl, and browser tools. You'll need a free Scrapfly API key first. Create one on the [Scrapfly signup page](https://scrapfly.io/register), then add it to `~/.hermes/.env`: bash```bash # ~/.hermes/.env SCRAPFLY_API_KEY=YOUR_SCRAPFLY_KEY ``` ### Path A: Install with `hermes skills install` (recommended) Run the Hermes-native install command from your terminal. Hermes fetches the skill from the public `scrapfly/skills` repository, runs a security scan, and installs it into `~/.hermes/skills/`: bash```bash hermes skills install scrapfly/skills/scraper --force hermes skills list # confirms: scrapfly-scraper ... enabled ``` You need the `--force` flag, and the reason is worth knowing. Hermes runs a security scan on every skill and returns a CAUTION verdict here. The findings are false positives for this skill. The scan flags `ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])` as possible data exfiltration, when the code only reads your own environment variable. It also flags `pip install scrapfly-sdk` as a supply-chain risk, when that's Scrapfly's official SDK. The repository ships five skills. Install whichever ones you need with the same pattern: - `scrapfly/skills/scraper`: fetch pages with anti-bot bypass and JavaScript rendering. - `scrapfly/skills/extraction`: pull structured data with prompts or templates. - `scrapfly/skills/screenshot`: capture full-page or element screenshots. - `scrapfly/skills/crawler`: crawl a domain recursively. - `scrapfly/skills/browser`: control a cloud browser for multi-step actions. Each skill is Python-SDK-based. Once installed, the agent reads the SKILL.md, runs `pip install scrapfly-sdk`, and writes short Python with `ScrapflyClient` and `ScrapeConfig`. There's no separate command-line step inside the skill. ### Path B: Connect the Scrapfly MCP server If you already run Hermes through MCP servers, add Scrapfly's hosted server instead. Put a `mcp_servers` block in `~/.hermes/config.yaml` and Hermes discovers its tools on startup: yaml```yaml # ~/.hermes/config.yaml mcp_servers: scrapfly: url: https://mcp.scrapfly.io/mcp headers: Authorization: "Bearer YOUR_SCRAPFLY_KEY" ``` The MCP path uses the [Scrapfly MCP server](https://scrapfly.io/docs/mcp/getting-started). The config shape follows Composio's published [ScrapingAnt-Hermes MCP integration](https://composio.dev/toolkits/scrapingant/framework/hermes-agent) pattern. The MCP path uses the [Scrapfly MCP server](https://scrapfly.io/docs/mcp/getting-started). The config shape follows Composio's published [ScrapingAnt-Hermes MCP integration](https://composio.dev/toolkits/scrapingant/framework/hermes-agent) pattern. [What Is MCP? Understanding the Model Context ProtocolWhat is MCP? Learn how the Model Context Protocol powers tools like Copilot Studio by giving AI models access to real-time, structured context.](https://scrapfly.io/blog/posts/what-is-mcp-understanding-the-model-context-protocol) ## Configure Scrapfly for Anti-Bot Bypass With the skill installed and your `SCRAPFLY_API_KEY` in `~/.hermes/.env`, the skill is already enabled. Confirm it with `hermes skills list`. You steer when the agent reaches for it with a routing rule in `~/.hermes/SOUL.md`. You turn on anti-bot bypass by telling the agent to use the SDK's `asp` and `render_js` options. There's no per-skill "default flags" config. The skill is a Markdown file the agent reads, so you steer behavior with instructions, not a config knob. Add a rule like this to `~/.hermes/SOUL.md`: text```text # ~/.hermes/SOUL.md When a fetched page returns less than 500 characters of content, or shows a verification or challenge screen, re-fetch it through the scrapfly-scraper skill with asp=True and render_js=True. ``` That rule teaches Hermes's self-improving loop to escalate to Scrapfly on its own. Under the hood, the skill writes a call like the one below. `asp=True` turns on anti-bot bypass, and `render_js=True` runs the page's JavaScript in a cloud browser: python```python from scrapfly import ScrapflyClient, ScrapeConfig import os client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"]) result = client.scrape(ScrapeConfig( url="https://web-scraping.dev/products", asp=True, # anti-bot bypass render_js=True, # run the page's JavaScript format="markdown", # return clean Markdown )) print(result.scrape_result["content"]) ``` A developer who prefers the MCP path gets the same capability through the `web_scrape` MCP tool instead of the skill. For the broader Scrapfly developer surface, see the [Scrape API docs](https://scrapfly.io/docs/scrape-api/getting-started). Scrapfly #### Need to bypass anti-bot protection? Scrapfly's Anti-Scraping Protection handles Cloudflare, DataDome, and more — automatically. [Try Free →](https://scrapfly.io/register)## When to Use Each Tool: Firecrawl vs Scrapling vs Scrapfly Keep Firecrawl for built-in `web_search` and lightweight `web_extract` on standard sites. Use Scrapling for Python-controlled local stealth and prototyping. Escalate to Scrapfly when you need cloud-managed anti-bot, multi-product fetch, or scale beyond a single machine. The matrix below positions all three by job. | Tool | Where it lives | Strongest at | Weakest at | Escalate when | |---|---|---|---|---| | Firecrawl | Hermes default web backend | Fast, simple, cloud-managed for easy sites | Rate limits, harder anti-bot sites, JS edge cases | Page returns a blank shell or you hit rate caps | | Scrapling | Optional local skill | Python stealth, adaptive parsing, free and local | Single-machine scale, your IP is the egress IP, upkeep | VPS IP gets blocked or scale outgrows one machine | | Scrapfly | Cloud fetch layer (skill or MCP) | 98% Cloudflare success, multi-product, 130M+ proxies in 190+ countries | Costs credits per request | Built-ins time out or you need screenshot, extract, or crawl | Most Hermes users running production workflows keep all three. Hermes routes by task type: - A simple search goes to Firecrawl-backed `web_extract`. - Python data wrangling goes to Scrapling. - Protected sites at scale go to Scrapfly. ## A Working Example: Scrape a Protected Page Through Hermes Once you install the skill, you can ask Hermes to scrape a protected page from any channel. That includes Telegram, Discord, the terminal, and the web UI. The agent routes the call through Scrapfly. Here's a one-line prompt you can paste into a Hermes chat: text```text Scrape the products page at https://web-scraping.dev/products through the Scrapfly skill and list the product names you find. ``` Behind the scenes, the run goes like this. Hermes routes the prompt to the `scrapfly-scraper` skill, or to the `web_scrape` MCP tool on Path B. Following the SKILL.md, the agent writes and runs the SDK call shown earlier, with `asp=True` and `render_js=True`. Scrapfly handles challenge solving, JavaScript rendering, and residential proxy routing in its cloud, then returns clean Markdown. Hermes parses that Markdown and replies with the product list. A real run against `web-scraping.dev/products` returned HTTP 200 and about 3,956 characters of Markdown. Here's a representative slice the agent reads back: markdown```markdown ![thumbnail of Box of Chocolate Candy](https://web-scraping.dev/assets/products/orange-chocolate-box-medium-1.webp) ### [Box of Chocolate Candy](https://web-scraping.dev/product/1) Indulge your sweet tooth with our Box of Chocolate Candy. Each box contains an assortment of rich, flavorful chocolates with a smooth, creamy filling. ``` The same install also gives you the `scrapfly-screenshot`, `scrapfly-extraction`, and `scrapfly-crawler` skills. One workspace then covers screenshots, structured extraction, and full crawls. JavaScript rendering plus anti-bot bypass costs more per request than a plain fetch. Check the [pricing page](https://scrapfly.io/pricing) for current rates. ## Scrapfly: The Cloud Fetch Layer for Hermes Agent ScrapFly's [Web Scraping API](https://scrapfly.io/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **99.99% success rate** across **130M+ proxies in 190+ countries**. - [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and JS-driven pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex). ## FAQ Is Scrapfly a replacement for Firecrawl in Hermes Agent?No, Scrapfly is an escalation layer, not a replacement. Keep Firecrawl for built-in `web_search` and lightweight `web_extract`, and reach for Scrapfly on protected sites, at scale, or when you need screenshots, extraction, or crawls. Can I use Scrapfly alongside the official Scrapling skill?Yes, both skills coexist in the same workspace. Scrapling handles local Python stealth for prototyping and single-machine jobs, while Scrapfly handles cloud production fetch, and you route between them with rules in `SOUL.md`. My Hermes agent runs on a VPS. Will Scrapfly fix the VPS IP-ban problem?Yes, because Scrapfly routes through a residential proxy mesh of 130M+ proxies across 190+ countries, so the target never sees your VPS IP. This is the failure VPS-hosted users keep reporting on r/hermesagent, local stealth can't help when the IP reputation itself is the problem. Do I need a separate API key if I have a Nous Tool Gateway subscription?Yes, you need a separate Scrapfly API key. The Nous Tool Gateway and Scrapfly are different services with separate billing, and you set both keys in `~/.hermes/.env`. How does this compare to my OpenClaw setup?The Scrapfly value is the same cloud fetch, but the install differs: OpenClaw uses ClawHub skills, while Hermes uses `hermes skills install` or an MCP server. If you run both side by side, each calls Scrapfly with the same API key. Is web scraping with Hermes Agent legal?Scraping publicly available data is generally lawful, but the rules depend on where you are, the site's terms, and the data you collect. Review the target's terms and get legal advice before scraping personal or copyrighted content. ## Summary Hermes Agent ships with a capable default stack: `web_search`, `web_extract`, browser automation, and the Scrapling skill. It handles easy and medium jobs well. The stack breaks on three points: the plain-HTTP fallback, anti-bot challenges that stall runs, and VPS IP bans. Scrapfly answers all three by separating the agent's logic from the bypass mechanics. Your Hermes agent plans tasks and owns memory on cheap hardware, and Scrapfly's cloud handles the protected-site fetch the defaults can't. You add it with `hermes skills install scrapfly/skills/scraper --force` or a `mcp_servers` block, then steer it from `SOUL.md`. Keep Firecrawl and Scrapling for what they do well, and add Scrapfly as the production layer for the cases that block them. Create a free key, install the scraper skill, and run the working example above against the most-blocked site in your current workflow. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [What Web Scraping Tools Does Hermes Agent Ship With?](#what-web-scraping-tools-does-hermes-agent-ship-with) - [Why the Default Stack Breaks on Protected Sites](#why-the-default-stack-breaks-on-protected-sites) - [How Scrapfly Fits Into the Hermes Stack](#how-scrapfly-fits-into-the-hermes-stack) - [Install the Scrapfly Skill in Hermes Agent](#install-the-scrapfly-skill-in-hermes-agent) - [Path A: Install with hermes skills install (recommended)](#path-a-install-with-hermes-skills-install-recommended) - [Path B: Connect the Scrapfly MCP server](#path-b-connect-the-scrapfly-mcp-server) - [Configure Scrapfly for Anti-Bot Bypass](#configure-scrapfly-for-anti-bot-bypass) - [When to Use Each Tool: Firecrawl vs Scrapling vs Scrapfly](#when-to-use-each-tool-firecrawl-vs-scrapling-vs-scrapfly) - [A Working Example: Scrape a Protected Page Through Hermes](#a-working-example-scrape-a-protected-page-through-hermes) - [Scrapfly: The Cloud Fetch Layer for Hermes Agent](#scrapfly-the-cloud-fetch-layer-for-hermes-agent) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Explore this Article with AI [ ChatGPT ](https://chat.openai.com/?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-hermes-agent) [ Gemini ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-hermes-agent) [ Grok ](https://x.com/i/grok?text=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-hermes-agent) [ Perplexity ](https://www.perplexity.ai/search/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-hermes-agent) [ Claude ](https://claude.ai/new?q=Summarize%20this%20page%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fweb-scraping-with-hermes-agent) ## Related Articles [ http nodejs ### Axios vs Fetch: Which HTTP Client to Choose in JS? Explore the differences between Fetch and Axios - two essential HTTP clients in JavaScript - and discover which is best ... ](https://scrapfly.io/blog/posts/axios-vs-fetch) [ python api ### Web Scraping for AI Agents in 2026 How AI agents consume the web, why their fetch layer breaks, and how to build agent-grade web access that holds up in pr... ](https://scrapfly.io/blog/posts/ai-agent-web-scraping) [ python api ### How to Build a Web Scraping Agent with Gemini Build a Gemini web scraping agent that works on real sites. Covers Gemini CLI skills, URL Context limits, Python pipelin... ](https://scrapfly.io/blog/posts/gemini-for-webscraping) ## Related Questions - [ Q How to add headers to every or some scrapy requests? ](https://scrapfly.io/blog/answers/how-to-add-headers-to-every-or-some-scrapy-requests) Bypass anti-bot protection automatically, **1,000 free credits** [Start Free](https://scrapfly.io/register)