[Blog](https://scrapfly.io/blog) / [ai](https://scrapfly.io/blog/tag/ai) / [Human-in-the-Loop (HITL) Cloud Browsers](https://scrapfly.io/blog/posts/human-in-the-loop-cloud-browsers) # Human-in-the-Loop (HITL) Cloud Browsers by [Hisham Medhat](https://scrapfly.io/blog/author/hisham) Jul 20, 2026 13 min read [\#ai](https://scrapfly.io/blog/tag/ai) [\#headless-browser](https://scrapfly.io/blog/tag/headless-browser) [\#playwright](https://scrapfly.io/blog/tag/playwright) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhuman-in-the-loop-cloud-browsers "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhuman-in-the-loop-cloud-browsers&text=Human-in-the-Loop%20%28HITL%29%20Cloud%20Browsers "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhuman-in-the-loop-cloud-browsers "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhuman-in-the-loop-cloud-browsers) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhuman-in-the-loop-cloud-browsers) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhuman-in-the-loop-cloud-browsers) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhuman-in-the-loop-cloud-browsers) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhuman-in-the-loop-cloud-browsers) Most browser automation doesn't fail on the easy steps. The script renders the page, fills the form, and clicks the right button. Then one CAPTCHA, MFA prompt, or manual review step blocks the whole run. Scrapfly cloud browsers let a person step into that exact moment, take over the live session, then hand it back. Below, you'll see when that handoff earns its cost and how to build it with a live Playwright demo on [web-scraping.dev/login](https://web-scraping.dev/login). [Web Scraping With Cloud BrowsersIntroduction cloud browsers and their benefits and a step-by-step setup with self-hosted Selenium-grid cloud browsers.](https://scrapfly.io/blog/posts/web-scraping-with-cloud-browsers) ## Key Takeaways Human-in-the-loop works best when you need session continuity more than raw speed. - Use HITL for rare blockers such as CAPTCHA pages, MFA prompts, or manual approval steps. - Keep the browser session alive with a persistent session ID, not a fresh browser launch. - Treat the human step as a handoff in the same session, not as a restart of the workflow. - Avoid HITL for high-volume, predictable jobs where human latency would dominate the run time. - Store the session ID and task state outside the browser so automation can reconnect cleanly. - Scrapfly's manual control flow fits best when you need real browser takeover, not a one-off challenge solver. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## What Is Human-in-the-Loop for Cloud Browsers? Human-in-the-loop, or HITL, means your automation can pause and let a human take over the same remote browser session. The human does the part machines shouldn't fake, then automation reconnects and keeps going. That difference matters. A CAPTCHA solver returns a token or an answer. A cloud browser handoff gives the human the full browser state: cookies, local storage, page history, and the exact tab where the workflow stopped. With Scrapfly's [Cloud Browser API](https://scrapfly.io/docs/cloud-browser-api/getting-started), the session becomes reusable. Connect with a fixed `session` value and set `auto_close=false`. The browser stays up the entire time. The script and the human just take turns driving it. The script automates the steps it can, then disconnects so a human can take over from the Scrapfly dashboard. When the human finishes and disconnects, the script reconnects to the same session, with the login and cookies they created still intact. python```python from playwright.sync_api import sync_playwright # 1. Connect and drive the easy part. with sync_playwright() as playwright: browser = playwright.chromium.connect_over_cdp(WS_URL) context = browser.contexts[0] page = context.pages[0] if context.pages else context.new_page() page.goto("https://web-scraping.dev/login", wait_until="domcontentloaded") page.fill("input[name='username']", "user123") page.fill("input[name='password']", "password") # 2. Release the session so a human can take over from the dashboard. browser.close() # 3. Wait for the human to attach and finish # (wait_for_human_cycle is defined under "How Scrapfly HITL Works"). wait_for_human_cycle(SESSION_ID) # 4. Reconnect to the same session and continue where the human left off. ``` This is the core HITL move. Use a persistent session, automate the easy part, then release the browser so a human can finish the manual step. After they disconnect, your script reconnects to the same session and continues with the state they created. The full runnable version is in the Live Example below. The pattern is simple: 1. Automation opens a persistent cloud browser session. 2. The workflow reaches a step that needs a human. 3. The script disconnects, releasing the session so a human can attach from the Scrapfly dashboard. 4. The human solves the blocker and disconnects. 5. Automation reconnects to the same session and continues with the state the human created. The handoff is what makes HITL useful. You don't build a second workflow for rare blockers; you keep one alive long enough for a human to unblock it. ## When to Use an HITL Cloud Browser HITL is a recovery pattern, not a default architecture. Add HITL when the blocker is rare, expensive to automate around, or impossible for software to solve honestly. The best fit usually looks like this: | Situation | Use HITL? | Why | |---|---|---| | Rare CAPTCHA or access review on a high-value workflow | Yes | A short handoff costs less than rebuilding the whole automation stack around edge cases. | | MFA, email approval, or phone verification | Yes | The human already owns the inbox, phone, or authenticator app. | | Account creation or login for a protected internal tool | Yes | Session continuity matters more than pure throughput. | | High-volume catalog scraping with predictable pages | No | Human latency will dominate the run time. | | Real-time alerting where the answer must arrive in seconds | No | Waiting for an operator defeats the purpose of the workflow. | | Stable test suites that never hit human-only checkpoints | No | Deterministic automation is cheaper and easier to maintain. | Three questions help you decide fast: - How often does the blocker appear? - How much value does one successful run create? - Do you need the exact session state after the blocker clears? If the blocker is rare, the run is valuable, and restarting would lose too much state, HITL is usually worth it. If the blocker happens on every request, HITL is the wrong tool. At that point, you need a different acquisition path, a different target, or a normal human-driven workflow. [Web Scraping With Cloud BrowsersIntroduction cloud browsers and their benefits and a step-by-step setup with self-hosted Selenium-grid cloud browsers.](https://scrapfly.io/blog/posts/web-scraping-with-cloud-browsers) The tradeoff gets sharper with AI agents. They reason well through most pages but still stop hard on human-only checkpoints. Scrapfly #### Need a cloud browser for scraping? Run headless browsers at scale with Scrapfly Cloud Browser — no infrastructure to manage. [Try Free →](https://scrapfly.io/register)## HITL for AI Agent Workflows AI agents make browser automation more flexible, but they don't remove hard blockers. A [browser agent](https://scrapfly.io/blog/posts/practical-guide-to-llm-agents) can infer the next click, summarize a page, and recover from minor UI changes. A browser agent still can't approve MFA from your phone or solve every novel verification step in a way you should trust. HITL matters more for agents than for fixed scripts. A deterministic scraper fails at one known boundary; an agent explores far deeper before it hits the one step that needs a person. In practice, the safe pattern looks like this: 1. Run the agent on a persistent browser session ID. 2. Let the orchestration layer detect a blocker such as a challenge page, repeated timeout, or approval step. 3. Keep the session open while a human takes over from the dashboard. 4. Let the human solve the blocker in the same tab. 5. Detect the resolution and resume the agent or switch to deterministic code for the final steps. The biggest gain is continuity. The human doesn't need to recreate the workflow from scratch, and the agent doesn't need to start over after the challenge clears. This also keeps responsibility in the right place. The agent handles navigation and repetitive page work. The human handles judgment, private credentials, and any step that should stay human. [Guide to Understanding and Developing LLM AgentsExplore how LLM agents transform AI, from text generators into dynamic decision-makers with tools like LangChain for automation, analysis & more!](https://scrapfly.io/blog/posts/practical-guide-to-llm-agents) The next step is the technical handoff itself: how to keep the session alive, expose it for manual control, and reconnect when the human finishes. ## How Scrapfly HITL Works Scrapfly exposes the pieces you need for a real handoff flow without inventing a new browser API. You connect through the normal CDP endpoint, add a stable session ID, and keep the session alive with `auto_close=false`. The browser stays running the whole time, and only one client controls it at once. Your script disconnects so a human can attach from the dashboard and solve the blocker. When the human disconnects, your script reconnects to the same session. The open tab does not carry over to the reconnecting client, but the cookies and storage do, so re-opening the page loads it in the state the human left behind. The [Human-in-the-Loop concept](https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop) centers the workflow around the Sessions dashboard. When a session is active, open the live screencast and take manual control alongside the CDP client. The same docs also show how to inspect sessions over HTTP. That lets your orchestration layer monitor session state from a separate process. Two endpoints are especially useful: - `GET https://browser.scrapfly.io/sessions?key=API_KEY` lists active sessions. - `GET https://browser.scrapfly.io/session/{session_id}?key=API_KEY` returns one session's state, including `attached_by` (who currently controls it). For cases where a separate process needs to check whether a human is still active, you can poll the session API: python```python import time import requests API_KEY = "YOUR_SCRAPFLY_API_KEY" def wait_for_human_cycle(session_id: str, timeout: int = 600) -> None: deadline = time.time() + timeout human_attached = False while time.time() < deadline: try: response = requests.get( f"https://browser.scrapfly.io/session/{session_id}", params={"key": API_KEY}, timeout=30, ) response.raise_for_status() except requests.RequestException: # The session API can return a brief 404 right after a disconnect. # Treat any transient error as "keep waiting", not a hard failure. time.sleep(2) continue attached_by = response.json().get("attached_by", "") if attached_by == "human_agent": human_attached = True elif human_attached and not attached_by: return # human attached and has now disconnected time.sleep(2) raise TimeoutError("Human takeover did not complete before the timeout") ``` This helper waits for a full takeover. It blocks until an operator attaches (`attached_by == "human_agent"`) and only returns once they disconnect (`attached_by == ""`). Call it after your script disconnects, then reconnect. A plain "wait until `attached_by` is empty" check is not enough, because the session is already empty in the moment between your disconnect and the human attaching. The core Scrapfly advantage for HITL is state survival. The session keeps its cookies and login state through the manual step, so a quick re-open lands you back where the human left off. ### Power your scraping with Scrapfly Forget about getting blocked. Scrapfly handles anti-bot bypasses, browser rendering, and proxy rotation so you can focus on the data. [Try for FREE!](https://scrapfly.io/register) ## Live Example: web-scraping.dev Login [web-scraping.dev](https://web-scraping.dev/) gives us a safe place to demonstrate the handoff mechanics. The site does not force a CAPTCHA on the login page, so this example uses a manual approval checkpoint instead. The script fills the login form and disconnects. A human submits it from the Scrapfly dashboard, then the script reconnects and confirms the authenticated state survived. First, install the dependencies: bash```bash pip install playwright playwright install chromium ``` Then run the handoff script: python```python import os import requests from playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeoutError API_KEY = os.environ["SCRAPFLY_API_KEY"] SESSION_ID = "web-scraping-login-demo" WS_URL = ( f"wss://browser.scrapfly.io?api_key={API_KEY}" f"&session={SESSION_ID}&auto_close=false&timeout=1800" "&proxy_pool=datacenter&os=linux" ) # 1. Automation drives the easy part. with sync_playwright() as playwright: browser = playwright.chromium.connect_over_cdp(WS_URL) context = browser.contexts[0] page = context.pages[0] if context.pages else context.new_page() page.goto("https://web-scraping.dev/login", wait_until="domcontentloaded") try: page.click("#cookie-ok", timeout=2_000) except PlaywrightTimeoutError: pass page.fill("input[name='username']", "user123") page.fill("input[name='password']", "password") print(f"Session ready: {SESSION_ID}") print("Open the session in the Scrapfly dashboard, click Submit, then disconnect.") # 2. Release the session so the human can take control. browser.close() # 3. Wait for the human to attach and finish. # wait_for_human_cycle() is defined above under "How Scrapfly HITL Works". wait_for_human_cycle(SESSION_ID) # 4. Reconnect to the same session and confirm the result. with sync_playwright() as playwright: browser = playwright.chromium.connect_over_cdp(WS_URL) context = browser.contexts[0] # The tab does not survive the handoff, but the session cookies do. # Re-open the page; the persisted auth cookie loads it already logged in. page = context.pages[0] if context.pages else context.new_page() page.goto("https://web-scraping.dev/login", wait_until="domcontentloaded") page.wait_for_selector("text=Logout", timeout=30_000) cookies = {cookie["name"]: cookie["value"] for cookie in context.cookies()} print("Login state confirmed:", "auth" in cookies) print("Current URL:", page.url) browser.close() # 5. Stop the session so it does not keep running (and billing). requests.post( f"https://browser.scrapfly.io/session/{SESSION_ID}/stop", params={"key": API_KEY}, timeout=30, ).raise_for_status() ``` After the human clicks Submit in the dashboard, the script prints: ``` Session ready: web-scraping-login-demo Open the session in the Scrapfly dashboard, click Submit, then disconnect. Login state confirmed: True Current URL: https://web-scraping.dev/login ``` This example shows the whole HITL loop across a handoff. Automation fills the login form, then disconnects so the human can submit it from the Scrapfly dashboard. Once the human disconnects, the script reconnects to the same session, detects the Logout link, and verifies that the `auth` cookie the human created survived. Because the session uses `auto_close=false`, its cookies and storage persist across the disconnect. The tab itself does not, so the script re-opens the page, which loads already logged in. Once that pattern works on a safe demo target, you can apply the same structure to real blockers. MFA, approval screens, and challenge pages all fit this flow. ## FAQ What is human-in-the-loop for cloud browsers?Human-in-the-loop lets a person take over the same remote browser session when automation gets stuck. After the manual step finishes, automation reconnects to the same session instead of starting over. When should I use HITL in browser automation?Use HITL when blockers are rare, the run is valuable, and losing session state would hurt more than waiting for a human. Avoid it for high-volume, predictable work. Does HITL preserve cookies and login state?Yes, if you keep the session alive and reconnect to the same browser session. That is the main reason to use HITL instead of restarting the workflow after the blocker clears. Can AI browser agents use HITL?Yes. Run the agent on a persistent session and let the orchestration layer detect the blocker. Then hand the session to a human before resuming. ## Summary Human-in-the-loop works because it treats manual intervention as a session handoff, not as an automation failure. The browser state stays alive. The human solves the one step machines shouldn't fake, and automation resumes from the exact point where it paused. HITL fits rare blockers such as CAPTCHA pages, MFA prompts, and internal review steps. It is the wrong fit for bulk scraping or any workflow where waiting on a human would become the main cost. With Scrapfly, persistent sessions preserve state, the dashboard provides manual control, and the session REST API reports who is attached. That keeps the handoff simple to build and easy to debug. The result is human recovery for cloud browser workflows, without a second automation stack to maintain. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [What Is Human-in-the-Loop for Cloud Browsers?](#what-is-human-in-the-loop-for-cloud-browsers) - [When to Use an HITL Cloud Browser](#when-to-use-an-hitl-cloud-browser) - [HITL for AI Agent Workflows](#hitl-for-ai-agent-workflows) - [How Scrapfly HITL Works](#how-scrapfly-hitl-works) - [Live Example: web-scraping.dev Login](#live-example-web-scraping-dev-login) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ blocking ### Best CAPTCHA Solving APIs in 2026 Compare the best CAPTCHA solving services including 2Captcha, CapSolver, Anti-Captcha, and browser-integrated solutions.... ](https://scrapfly.io/blog/posts/best-captcha-solving-api) [ api headless-browser ### Best Browser Automation Tools in 2026 Explore the best browser automation tools of 2026. This complete guide compares Cloud APIs, Open-Source frameworks, AI A... ](https://scrapfly.io/blog/posts/best-browser-automation-tools) [ blocking api ### Web Scraping with Hermes Agent Hermes Agent ships with Firecrawl and Scrapling, but they break on protected sites. Add Scrapfly as a cloud fetch layer ... ](https://scrapfly.io/blog/posts/web-scraping-with-hermes-agent) ## Related Questions - [ Q How to take screenshots in NodeJS? ](https://scrapfly.io/blog/answers/how-to-take-screenshots-nodejs) Run headless browsers at scale, **1,000 free credits** [Start Free](https://scrapfly.io/register)