[Blog](https://scrapfly.io/blog) / [headless-browser](https://scrapfly.io/blog/tag/headless-browser) / [How to Scrape Infinite Scroll, Load More & Paginated Pages](https://scrapfly.io/blog/posts/how-to-scrape-infinite-scroll-load-more-and-paginated-pages) # How to Scrape Infinite Scroll, Load More & Paginated Pages by [Hisham Medhat](https://scrapfly.io/blog/author/hisham) Jul 20, 2026 17 min read [\#headless-browser](https://scrapfly.io/blog/tag/headless-browser) [\#hidden-api](https://scrapfly.io/blog/tag/hidden-api) [\#python](https://scrapfly.io/blog/tag/python) [\#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide) [ ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-infinite-scroll-load-more-and-paginated-pages "Share on LinkedIn") [ ](https://x.com/intent/tweet?url=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-infinite-scroll-load-more-and-paginated-pages&text=How%20to%20Scrape%20Infinite%20Scroll%2C%20Load%20More%20%26%20Paginated%20Pages "Share on X") [ ](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-infinite-scroll-load-more-and-paginated-pages "Share on Facebook") Summarize this article with [ ](https://chat.openai.com/?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-infinite-scroll-load-more-and-paginated-pages) [ ](https://claude.ai/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-infinite-scroll-load-more-and-paginated-pages) [ ](https://x.com/i/grok?text=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-infinite-scroll-load-more-and-paginated-pages) [ ](https://www.perplexity.ai/search/new?q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-infinite-scroll-load-more-and-paginated-pages) [ ](https://www.google.com/search?udm=50&aep=11&q=Summarize%20this%20article%20and%20explain%20how%20Scrapfly%20helps%20me%20scrape%20any%20website%20at%20scale%20and%20bypass%20anti-bot%20systems%20for%20my%20use%20case%3A%20https%3A%2F%2Fscrapfly.io%2Fblog%2Fposts%2Fhow-to-scrape-infinite-scroll-load-more-and-paginated-pages) **Web Scraping API**Scrape any website with anti-bot bypass, proxy rotation, and JS rendering. [ Learn More ](https://scrapfly.io/products/web-scraping-api) [ Docs ](https://scrapfly.io/docs/scrape-api/getting-started) You wrote the scraper, it parsed the page cleanly, and it handed back the first ten rows out of a few hundred. The list keeps growing as you scroll in a real browser, yet your HTTP request never sees past that first batch. The cause is always the same. The initial HTML holds only the first batch; the rest loads through follow-up requests. Spot the pattern, then reverse-engineer that request when you can and fall back to a headless browser only when you can't. [How to Scrape Hidden APIsIn this tutorial we'll be taking a look at scraping hidden APIs which are becoming more and more common in modern dynamic websites - what's the best way to scrape them?](https://scrapfly.io/blog/posts/how-to-scrape-hidden-apis) ## Key Takeaways - **The initial HTML holds only the first batch**; the rest loads through follow-up requests. - **Classify the pattern in the DevTools Network tab** before writing any scraping code. - **Reverse-engineer the request first**; it beats a browser on speed and cost. - **Cursor pagination powers most infinite scroll**; read the token, you can't guess it. - **Use a headless browser only as a fallback**, with a stop condition and guard. - **Anti-bot guards the backend API too**, so the real work at scale is staying unblocked. - **At scale, Scrapfly's Web Scraping API** handles blocking, so you keep the paging loop. **Get web scraping tips in your inbox**Trusted by 100K+ developers and 30K+ enterprises. Unsubscribe anytime. ## Why Your Scraper Only Returns the First Page Your scraper returns only the first page for one reason. The server puts a small first batch in the initial HTML, then loads the rest through separate requests. Your single request grabs that first document and never fires the follow-up calls. Those follow-up calls return JSON or HTML fragments over XHR or Fetch. They're the background requests a page sends after the first load. Everything past batch one sits behind a request you haven't made yet. Sites split content this way for speed. A page that ships 20 rows loads faster than one shipping 2,000. It feels quicker on mobile and saves bandwidth until the reader asks for more (LogRocket, 2024). Three actions trigger that next request: scrolling to the bottom (infinite scroll), clicking a button ("Load More"), or changing a URL parameter (classic pagination). Each maps to the same idea, a request for the next slice of data. Developers on r/nocode regularly report that general-purpose scrapers break when a site loads data dynamically (r/nocode, 2025). The fix isn't a heavier tool. It's knowing which request to make, which the next section helps you do fast. ## How to Tell Which Loading Pattern a Page Uses Open your browser's DevTools Network tab, filter to Fetch/XHR, then scroll or click and watch what fires. The request that appears when new rows show up is the one you need to replicate. Reading its URL and query parameters tells you the pattern in seconds. Before you touch code, learn the five patterns you will meet. For a hands-on place to practice this exact diagnosis, the [endless paging playground](https://scrapfly.io/scrapeground/paging/endless) runs a live infinite-scroll feed you can inspect. ### The Five Pagination Patterns Every paginated, load-more, or infinite-scroll page uses one of these five patterns: - **Numbered / URL pagination**: the page number lives in the URL, like `/reviews?page=3` or `/reviews/page/3`. Each page is its own document. - **Offset and limit**: the request asks for a window of rows, like `?offset=40&limit=20`. You increment the offset to walk the list. - **Cursor-based**: the response returns a token pointing at the next batch, like `?cursor=abc123`. You cannot guess it; you read it from each response. - **"Load More" button**: a click fires the next-batch request. The backend call is usually identical to infinite scroll; only the trigger differs. - **Infinite scroll**: scrolling near the bottom fires the next-batch request automatically. Most modern infinite scroll uses cursor pagination underneath. Load more and infinite scroll are twins. They share one backend request and differ only in the front-end trigger, so a fix for one usually works for the other. ### Diagnosing the Pattern in 60 Seconds Start by disabling JavaScript in DevTools and reloading. Whatever survives is server-rendered, and that confirms your first batch. Turn JavaScript back on, open the Network tab, filter to Fetch/XHR, and clear it. Now scroll or click once. A new request appears. Read its URL: a `page` or `offset` parameter means numbered or offset pagination, while `cursor`, `after`, or `next` means cursor-based. That single request is your target for the faster path. The DevTools Network tab is the core skill here, and it repays a closer look. Our guide to [browser developer tools in web scraping](https://scrapfly.io/blog/answers/browser-developer-tools-in-web-scraping) covers reading requests, headers, and payloads in full. With the pattern identified, you can decide how to capture every batch, starting with the cheapest option. ## Reverse-Engineer the Pagination API First If you can find and replicate the request the page makes for the next batch, do that first. Calling the endpoint directly skips rendering, so it's faster, cheaper, and less fragile than automation. It's the default this guide recommends. The recipe is the same for every pattern. Read the request from the Network tab, then copy its URL, method, and headers. Loop the page, offset, or cursor parameter until the response is empty. Many endpoints check a `Referer` header, so copy that too. ### Offset and Page-Number Pagination For offset or page-number pagination, increment the parameter and stop when a page returns nothing or errors. The `web-scraping.dev/testimonials` feed loads batches from `/api/testimonials?page=N`. That call needs a `Referer` header you spot only by reading it. Here is a plain `requests` loop that collects every page and stops on the first empty or failed response: python```python import requests from parsel import Selector session = requests.Session() session.headers["Referer"] = "https://web-scraping.dev/testimonials" all_testimonials = [] page = 1 while True: response = session.get(f"https://web-scraping.dev/api/testimonials?page={page}") if response.status_code != 200: break # server rejects pages past the last one texts = Selector(response.text).css("p.text::text").getall() if not texts: break # empty batch means we ran off the end all_testimonials.extend(t.strip() for t in texts) page += 1 print(f"collected {len(all_testimonials)} testimonials across {page - 1} pages") ``` The loop has two stop conditions: a non-200 status and an empty batch. Either one ends it cleanly, so you never guess how many pages exist. Running it returns every row without a browser: text```text collected 60 testimonials across 6 pages ``` This tool-agnostic pattern works in any language with an HTTP client. Offset pagination is the easy case because you control the position; cursor pagination hands that control to the server. ### Cursor-Based Pagination With cursor-based pagination, the server returns a token pointing at the next batch, and you pass it back next time. You can't predict the token, so read it from each response and stop when it's empty. This powers most infinite scroll you'll meet. The endpoint below is a representative example, not a real one, since cursor APIs vary by site: python```python import requests session = requests.Session() all_items = [] cursor = None while True: params = {"limit": 20} if cursor: params["cursor"] = cursor # illustrative endpoint - read the real one from your DevTools Network tab data = session.get("https://api.example.com/v1/items", params=params).json() all_items.extend(data["items"]) cursor = data.get("next_cursor") if not cursor: # null or missing token means the last page break print(f"collected {len(all_items)} items") ``` The stop condition is the missing token, not a page count. You keep asking until the server stops handing you a cursor. Because you cannot skip ahead, cursor pagination is strictly sequential, which matters when you plan concurrency later. ### Replicating the Request and Knowing When to Stop Replicating a request means matching its URL, method, headers, and body closely enough that the server answers. The header that trips people up most is `Referer`, then tokens like an `x-csrf-token` or an API key from the original request. Copy every header the Network tab shows, then remove them one by one to find which are truly required. Give every loop a clear stop condition. Stop on an empty batch, a repeated set of IDs, or an HTTP error, and never trust the page to end the loop for you. Batches can also overlap, so de-duplicate on a stable item ID before you count results. When the request itself is hidden behind a browser handshake, our guide to [capturing background requests with headless browsers](https://scrapfly.io/blog/posts/web-scraping-background-requests-with-headless-browsers-and-python) shows how to record it once, then replay it with a plain HTTP client. When that fails, the browser becomes the tool, not the scout. [How to Scrape Hidden Web DataThe visible HTML doesn't always represent the whole dataset available on the page. In this article, we'll be taking a look at scraping of hidden web data. What is it and how can we scrape it using Python?](https://scrapfly.io/blog/posts/how-to-scrape-hidden-web-data) ## Run a Headless Browser When You Cannot Hit the API When you cannot replicate the request, render the page in a headless browser and reproduce the user action. Signed tokens, heavy obfuscation, or a WAF on the endpoint can make reverse-engineering impractical. In those cases a browser loads the page, runs its JavaScript, and lets the site's own code fire the requests for you. Be honest about the cost. This path is slower and heavier than an HTTP loop, so treat it as the fallback, not the default. The examples below use [Playwright](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python); the same logic applies to [Selenium](https://scrapfly.io/blog/posts/web-scraping-with-selenium-and-python) or [Puppeteer](https://scrapfly.io/blog/posts/web-scraping-with-puppeteer-and-nodejs). Those guides cover install and setup, so this section skips it. ### Infinite Scroll: Scroll Loops With a Stop Condition For infinite scroll, scroll toward the bottom in a loop, wait for new rows, and repeat until the count stops growing. The key detail is a stop condition paired with a max-iteration guard, so a spinner-only page can't trap your loop. python```python from playwright.sync_api import sync_playwright with sync_playwright() as pw: browser = pw.chromium.launch(headless=True) page = browser.new_page() page.goto("https://web-scraping.dev/testimonials", wait_until="networkidle") previous_count = 0 for _ in range(15): # max-iteration guard: never loop forever page.mouse.wheel(0, 10000) try: page.wait_for_function( "count => document.querySelectorAll('.testimonial').length > count", arg=previous_count, timeout=3000, ) except Exception: break # no new items arrived: we hit the bottom previous_count = page.locator(".testimonial").count() count = page.locator(".testimonial").count() browser.close() print(f"loaded {count} testimonials after scrolling") ``` Notice the wait strategy. Instead of a fixed `sleep`, the loop waits for the item count to grow, which is faster and less flaky. The guard caps the loop at 15 scrolls, and the run collects the full feed: text```text loaded 60 testimonials after scrolling ``` That same wait-then-check shape drives the button variant, with a click standing in for the scroll. ### "Load More" Buttons: Click-and-Wait Loops For a "Load More" button, locate it, click, wait for the new batch, and repeat until the button disappears or stops adding rows. The snippet below is illustrative, since selectors differ per site, but the control flow is exactly what you will write: python```python # illustrative selectors - replace with the ones from your target page while page.locator("button.load-more").count() > 0: before = page.locator(".item").count() page.locator("button.load-more").click() try: page.wait_for_function( "n => document.querySelectorAll('.item').length > n", arg=before, timeout=5000, ) except Exception: break # button did nothing: stop ``` The button vanishing is your natural stop condition, backed by the same count check in case it lingers after the last batch. Numbered pagination in a browser follows the identical idea. ### Numbered and Next-Button Pagination in the Browser For numbered or next-button pagination, click "Next" or follow the next link until it's gone. Read the control's state each iteration, and stop when it's missing or disabled. Add a max-page guard too, since a broken site can loop "Next" in place. With both methods on the table, the real question is which one to reach for first, which comes down to a short set of tradeoffs. Scrapfly #### Need a cloud browser for scraping? Run headless browsers at scale with Scrapfly Cloud Browser — no infrastructure to manage. [Try Free →](https://scrapfly.io/register)## Which Method Should You Use? Try to reverse-engineer the request first, and fall back to a headless browser only when you can't replicate it. The API path wins on speed and cost; the browser path wins on resilience against obfuscation. The table below shows the tradeoffs. | | Reverse-engineer the API | Headless browser | |---|---|---| | **Speed & cost** | Fast and cheap; no rendering, one request per batch | Slow and heavy; a full browser per page load | | **Reliability** | Stable while the endpoint and its params hold | Survives obfuscation and signed tokens the API path cannot | | **When it fails** | Signed tokens, obfuscated params, or a WAF on the endpoint | Aggressive anti-bot that fingerprints the browser itself | | **Best for** | Numbered, offset, and cursor APIs you can read and replay | Pages whose next-batch request you genuinely cannot reproduce | The rule of thumb is short. Read the request first; render the page only when the request refuses to be read. Once either method works locally, the next challenge is running it at volume without getting blocked. ## How to Scrape Paginated Content at Scale At scale the hard part isn't the loop; it's staying unblocked, capping concurrency, and being able to resume. And anti-bot protection guards the backend API too, not only the page, a point most guides skip. The JSON endpoint you reverse-engineered often sits behind the same defenses as the site. Hammering it gets you blocked as quickly as the page would. Control concurrency deliberately. Parallelize across page ranges or independent cursors, but cap the in-flight requests and add delays, since a paginated endpoint rate-limits per IP quickly. Cursor pagination is sequential by nature, so parallelize across separate starting points, not within one cursor chain. Build in resumability. Checkpoint the last page or cursor you finished, so a run that dies at page 400 restarts there instead of refetching everything. A stable item ID lets you drop the duplicates that overlap across a restart. A managed fetch layer lets you keep your reverse-engineering loop while it handles the blocking problem. You can point it straight at the discovered JSON endpoint, or let it render JavaScript and scroll the feed for the browser path, all through one API. ScrapFly's [Web Scraping API](https://scrapfly.io/web-scraping-api) is a single HTTP endpoint for collecting web data at scale, with a **99.99% success rate** across **130M+ proxies in 190+ countries**. - [Anti-Scraping Protection bypass](https://scrapfly.io/docs/scrape-api/anti-scraping-protection) - automatically defeats Cloudflare, DataDome, PerimeterX, Akamai, and 90+ other bot systems. - [Smart proxy rotation](https://scrapfly.io/docs/scrape-api/proxy) - residential and datacenter pools with country and ASN level geo-targeting. - [JavaScript rendering](https://scrapfly.io/docs/scrape-api/javascript-rendering) - render SPAs and dynamic pages through real cloud browsers. - [Browser automation scenarios](https://scrapfly.io/docs/scrape-api/javascript-scenario) - scroll, click, fill forms, and wait for elements without managing a browser fleet. - [Format conversion](https://scrapfly.io/docs/scrape-api/getting-started#api_param_format) - return pages as HTML, JSON, clean text, or LLM ready Markdown. - [Session management](https://scrapfly.io/docs/scrape-api/session) - keep cookies, headers, and IPs consistent across multi step flows. - [Smart caching](https://scrapfly.io/docs/scrape-api/getting-started#api_param_cache) - cache successful responses to cut cost on repeat scraping jobs. - [Python](https://scrapfly.io/docs/sdk/python), [TypeScript](https://scrapfly.io/docs/sdk/typescript), [Scrapy](https://scrapfly.io/docs/sdk/scrapy), and [no-code integrations](https://scrapfly.io/docs/integration/getting-started) including [Make](https://scrapfly.io/integration/make), [n8n](https://scrapfly.io/integration/n8n), [Zapier](https://scrapfly.io/integration/zapier), [LangChain](https://scrapfly.io/integration/langchain), and [LlamaIndex](https://scrapfly.io/integration/llamaindex). For the API path, point the same endpoint at your discovered URL and turn on `asp` to pass anti-bot. Add the `Referer` header the endpoint expects: python```python from scrapfly import ScrapflyClient, ScrapeConfig from parsel import Selector client = ScrapflyClient(key="YOUR_SCRAPFLY_KEY") result = client.scrape(ScrapeConfig( url="https://web-scraping.dev/api/testimonials?page=1", headers={"Referer": "https://web-scraping.dev/testimonials"}, asp=True, )) count = len(Selector(result.scrape_result["content"]).css("p.text").getall()) print(f"page 1 returned {count} testimonials, asp handled blocking") ``` For the browser path, set `render_js=True` and `auto_scroll=True` to load an infinite feed without writing a scroll loop yourself. The `rendering_wait` gives late batches time to arrive: python```python from scrapfly import ScrapflyClient, ScrapeConfig from parsel import Selector client = ScrapflyClient(key="YOUR_SCRAPFLY_KEY") result = client.scrape(ScrapeConfig( url="https://web-scraping.dev/testimonials", render_js=True, auto_scroll=True, rendering_wait=3000, asp=True, )) count = len(Selector(result.scrape_result["content"]).css(".testimonial").getall()) print(f"auto-scroll loaded {count} testimonials") ``` Both variants run against the same sandbox and return the expected counts: text```text page 1 returned 10 testimonials, asp handled blocking auto-scroll loaded 60 testimonials ``` For finer control you can pass a `js_scenario`, a list of `scroll`, `click`, and `wait_for_selector` steps, when a page needs an action first. To follow next-page links across a whole domain, the [Crawler API](https://scrapfly.io/crawler-api) handles that traversal. ### Web Scraping API Scrape any website with our powerful API. Anti-bot bypass, JavaScript rendering, and rotating proxies built-in. [Try Web Scraping API](https://scrapfly.io/docs/scrape-api/getting-started) ## Common Pitfalls When Scraping Paginated Pages Most paginated scrapes fail in a handful of predictable ways. Match your symptom to this list and the fix is usually one line: - **Only the first page comes back**: you parsed the initial HTML and never fired the follow-up request. Find the XHR in the Network tab or increment the page parameter. - **The loop never ends**: no stop condition, so the page keeps returning the last batch or a spinner. Stop on an empty batch, repeated IDs, or a max-iteration guard. - **Duplicate or overlapping rows**: batches overlap or re-emit items across requests. De-duplicate on a stable item ID before counting. - **Off-by-one page ranges**: you started at page 0 instead of 1, or dropped the final partial page. Verify the first and last page by hand. - **Blocked mid-run**: the endpoint rate-limits you after N requests. Slow down, rotate identity, or route through an anti-bot-aware fetch layer. That last failure is the one that scales worst, which is why blocking, not the loop, is what breaks large scrapes. With the pitfalls mapped, a few common questions remain. ## FAQ Do I always need a headless browser to scrape infinite scroll?No. Often you can find and replicate the request the page makes for the next batch, usually in the DevTools Network tab. Then a plain HTTP client is faster and steadier. Reach for a headless browser only when the request can't be replicated. What is the difference between offset and cursor-based pagination?Offset pagination asks for a position, like `?offset=40&limit=20`, so you can jump to any window. Cursor pagination passes a token that points at the next batch, so you must read the token from each response and cannot skip ahead. How do I stop an infinite-scroll loop from running forever?Stop when the item count stops growing, when a batch returns zero new items, or when you hit a preset max-iteration guard. Never rely on the page to end the loop for you. Can I scrape infinite scroll without Python?Yes. The same two methods apply in any language: replicate the request with any HTTP client, or run a browser with Playwright or Puppeteer in JavaScript. Is it legal to scrape paginated content?Scraping public data is generally legal, but the rules depend on the site's terms, the data type, and your local laws. Avoid login-gated or paywalled content, and review the site's terms before you scrape. ## Summary Your scraper stops at the first page for a structural reason, not a parser bug. The server ships a small first batch, then loads the rest through follow-up requests that a scroll, click, or page parameter triggers. See that split and the rest follows. Identify the pattern in the DevTools Network tab, then reverse-engineer the request whenever you can. Hitting the endpoint directly beats a browser on speed, cost, and stability. Fall back to a headless browser only when signed tokens or a WAF block the replay. Give every loop a stop condition and a max-iteration guard. At scale the loop is the easy part; staying unblocked is the work. A managed layer like Scrapfly's Web Scraping API keeps your reverse-engineering loop while handling JavaScript rendering and anti-bot. For smaller jobs, a well-behaved plain-HTTP or Playwright scraper stays perfectly viable. Legal Disclaimer and PrecautionsThis tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect: - Do not scrape at rates that could damage the website. - Do not scrape data that's not available publicly. - Do not store PII of EU citizens protected by GDPR. - Do not repurpose *entire* public datasets which can be illegal in some countries. Scrapfly does not offer legal advice but these are good general rules to follow. For more you should consult a lawyer. [ Add as a preferred source ](https://google.com/preferences/source?q=scrapfly.io) Table of Contents Table of Contents- [Key Takeaways](#key-takeaways) - [Why Your Scraper Only Returns the First Page](#why-your-scraper-only-returns-the-first-page) - [How to Tell Which Loading Pattern a Page Uses](#how-to-tell-which-loading-pattern-a-page-uses) - [The Five Pagination Patterns](#the-five-pagination-patterns) - [Diagnosing the Pattern in 60 Seconds](#diagnosing-the-pattern-in-60-seconds) - [Reverse-Engineer the Pagination API First](#reverse-engineer-the-pagination-api-first) - [Offset and Page-Number Pagination](#offset-and-page-number-pagination) - [Cursor-Based Pagination](#cursor-based-pagination) - [Replicating the Request and Knowing When to Stop](#replicating-the-request-and-knowing-when-to-stop) - [Run a Headless Browser When You Cannot Hit the API](#run-a-headless-browser-when-you-cannot-hit-the-api) - [Infinite Scroll: Scroll Loops With a Stop Condition](#infinite-scroll-scroll-loops-with-a-stop-condition) - ["Load More" Buttons: Click-and-Wait Loops](#load-more-buttons-click-and-wait-loops) - [Numbered and Next-Button Pagination in the Browser](#numbered-and-next-button-pagination-in-the-browser) - [Which Method Should You Use?](#which-method-should-you-use) - [How to Scrape Paginated Content at Scale](#how-to-scrape-paginated-content-at-scale) - [Common Pitfalls When Scraping Paginated Pages](#common-pitfalls-when-scraping-paginated-pages) - [FAQ](#faq) - [Summary](#summary) Join the Newsletter Get monthly web scraping insights Scale Your Web Scraping Anti-bot bypass, browser rendering, and rotating proxies, all in one API. Start with 1,000 free credits. No credit card required 1,000 free API credits Anti-bot bypass included [Start Free](https://scrapfly.io/register) [View Docs](https://scrapfly.io/docs/onboarding) Not ready? Get our newsletter instead. ## Related Articles [ python headless-browser ### How to Scrape Dynamic Websites Using Headless Web Browsers Introduction to using web automation tools such as Puppeteer, Playwright, Selenium and ScrapFly to render dynamic websit... ](https://scrapfly.io/blog/posts/scraping-using-browsers) [ python seo ### How to Scrape Google Trends using Python In this article we'll be taking a look at scraping Google Trends - what it is and how to scrape it? For this example, we... ](https://scrapfly.io/blog/posts/how-to-scrape-google-trends) [ python hidden-api ### How to Scrape YouTube in 2026 Learn how to scrape YouTube channel, video, comment, and Shorts data in Python using hidden APIs and yt-dlp. No API key ... ](https://scrapfly.io/blog/posts/how-to-scrape-youtube) Run headless browsers at scale, **1,000 free credits** [Start Free](https://scrapfly.io/register)