Prerequisites

Before getting started, make sure you have the following:

• n8n instance (cloud or self-hosted)
• Your Scrapfly API key (create account)

Setup Instructions

Integrate Scrapfly into your n8n workflows using HTTP Request nodes. This enables powerful automated web scraping pipelines.

1. Create New n8n Workflow

   Start a new workflow in n8n:

   1. Log in to your n8n instance
   2. Click "New Workflow"
   3. Add a trigger node (Schedule, Webhook, Manual, etc.)
   Tip: Common Trigger Nodes

   • Schedule Trigger: Run scraping on a schedule (hourly, daily, etc.)
   • Webhook: Trigger scraping via HTTP request
   • Manual Trigger: Run scraping on demand
   • Email Trigger: Scrape URLs from incoming emails
2. Add HTTP Request Node for Scrapfly

   Configure an HTTP Request node to call the Scrapfly API:

   1. Click the "+" button to add a new node
   2. Search for and select "HTTP Request"
   3. Configure the node with these settings:

   HTTP Request Configuration:

   • Method: GET
   • URL: https://api.scrapfly.io/scrape
   • Query Parameters:
     • key: __API_KEY__
     • url: {{ $json.url }} (or hardcoded URL)
     • format: markdown
   • Response Format: JSON
   Sign up for free to get your API key.
   Tip: Using Credentials

   Instead of hardcoding the API key, create an n8n credential:

   • Go to Credentials → Create New
   • Choose "Header Auth"
   • Add parameter: X-API-Key with your Scrapfly API key
   • Select this credential in the HTTP Request node
3. Process Scrapfly Response

   Extract and process the scraped content using n8n's data transformation nodes:

   Extract Content with Set Node:

   1. Add a "Set" node after the HTTP Request
   2. Configure it to extract the content:
   • Field Name: content
   • Value: {{ $json.result.content }}

   Alternative: Use Code Node for Complex Processing:

   // Extract and process scraped content
   const scrapedData = $input.all();
   
   return scrapedData.map(item => ({
     json: {
       url: item.json.result.url,
       content: item.json.result.content,
       title: item.json.result.content.split("\n")[0], // Extract first line as title
       scrapedAt: new Date().toISOString()
     }
   }));
   Pro Tip: Use the Code node to clean, parse, or transform scraped data before saving it!
4. Add Output Action

   Send the scraped data to your desired destination:

   Popular Output Nodes:

   • Google Sheets: Save data to a spreadsheet
   • Airtable: Store in a database
   • Slack/Discord: Send alerts or notifications
   • Email: Email reports
   • Webhook: Send to your custom API
   • File: Save as JSON, CSV, or other formats
   Example: Save to Google Sheets

   Add a Google Sheets node with these settings:

   • Operation: Append Row
   • Spreadsheet: Select your sheet
   • Columns: Map url, content, scrapedAt
5. Test and Activate Workflow

   Test your workflow and activate it for production use:

   1. Click "Execute Workflow" to test manually
   2. Review the output from each node
   3. Once working correctly, toggle the workflow to "Active"
   4. Monitor executions in the "Executions" tab
   Tip: Error Handling

   Add error handling to your workflow:

   • Enable "Continue On Fail" on HTTP Request node
   • Add an IF node to check for errors: {{ $json.error }}
   • Route errors to a notification or logging node

Example Workflow Templates

Troubleshooting

Alternative Automation Platforms

While n8n offers powerful open-source automation, you might also want to explore these no-code automation alternatives for Scrapfly integration:

Next Steps

• Explore available MCP tools and their capabilities
• See real-world examples of what you can build
• Learn about authentication methods in detail
• Read the FAQ for common questions