> For the complete documentation index, see [llms.txt](https://docs.agenticflow.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.agenticflow.ai/reference/nodes/web_scraping_apify.md).

# Web Scraping using Apify

**Action ID:** `web_scraping_apify`

## Description

Web Scraping using Apify

## Connection

| Name               | Description                                 | Required | Category |
| ------------------ | ------------------------------------------- | -------- | -------- |
| PixelML Connection | The PixelML connection to call PixelML API. | True     | pixelml  |

## Input Parameters

| Name                  | Type    | Required | Default            | Description                                                 |
| --------------------- | ------- | :------: | ------------------ | ----------------------------------------------------------- |
| web\_urls             | array   |     ✓    | -                  | List of web URLs to scrape (1-10 URLs)                      |
| crawler\_type         | string  |     -    | playwright:firefox | Crawling engine (firefox, chrome, adaptive, cheerio, jsdom) |
| max\_tokens\_per\_url | integer |     -    | 32000              | Maximum tokens to scrape per URL (1-64000)                  |

<details>

<summary>View JSON Schema</summary>

```json
{
  "description": "Web scraping node input.",
  "properties": {
    "web_urls": {
      "description": "List of Web URL to scrape.",
      "items": {
        "type": "string"
      },
      "maxItems": 10,
      "minItems": 1,
      "title": "Web URLs",
      "type": "array"
    },
    "crawler_type": {
      "default": "playwright:firefox",
      "description": "Crawling engine to use. Default is playwright:firefox.",
      "enum": [
        "playwright:firefox",
        "playwright:chrome",
        "playwright:adaptive",
        "cheerio",
        "jsdom"
      ],
      "title": "Crawling Engine",
      "type": "string"
    },
    "max_tokens_per_url": {
      "default": 32000,
      "description": "Maximum number of tokens to scrape per URL. Default is 10000.",
      "maximum": 64000,
      "minimum": 1,
      "title": "Max Tokens per URL",
      "type": "integer"
    }
  },
  "required": [
    "web_urls"
  ],
  "title": "ApifyWebScrapingNodeInput",
  "type": "object"
}
```

</details>

## Output Parameters

| Name              | Type  | Description                                     |
| ----------------- | ----- | ----------------------------------------------- |
| scraped\_contents | array | List of extracted content strings from each URL |

<details>

<summary>View JSON Schema</summary>

```json
{
  "description": "Web scraping node output.",
  "properties": {
    "scraped_contents": {
      "description": "The extracted content from the web URLs.",
      "items": {
        "type": "string"
      },
      "title": "List of extracted content",
      "type": "array"
    }
  },
  "required": [
    "scraped_contents"
  ],
  "title": "ApifyWebScrapingNodeOutput",
  "type": "object"
}
```

</details>

## How It Works

This node uses Apify's web scraping infrastructure through the PixelML API to extract content from web pages. You provide URLs and select a crawling engine. The node uses headless browsers (Playwright) or HTML parsers (Cheerio/JSDOM) to access pages, extract text content, and return it in a structured format. The max\_tokens parameter limits the amount of content extracted per URL, useful for controlling costs and processing time.

## Usage Examples

### Example 1: Scrape Blog Articles

**Input:**

```
web_urls: ["https://example.com/blog/article-1", "https://example.com/blog/article-2"]
crawler_type: "playwright:firefox"
max_tokens_per_url: 32000
```

**Output:**

```
scraped_contents: [
  "Article 1 title and content extracted here...",
  "Article 2 title and content extracted here..."
]
```

### Example 2: Fast Scraping with Cheerio

**Input:**

```
web_urls: ["https://news.example.com/latest"]
crawler_type: "cheerio"
max_tokens_per_url: 10000
```

**Output:**

```
scraped_contents: [
  "Latest news article content with text only..."
]
```

### Example 3: JavaScript-Heavy Site

**Input:**

```
web_urls: ["https://spa-app.example.com/data"]
crawler_type: "playwright:adaptive"
max_tokens_per_url: 20000
```

**Output:**

```
scraped_contents: [
  "Dynamically loaded content from JavaScript app..."
]
```

## Common Use Cases

* **Content Aggregation**: Collect articles, blog posts, or news from multiple sources
* **Competitive Analysis**: Extract product information, pricing, or features from competitor websites
* **Data Mining**: Gather data from public websites for research or analysis
* **SEO Analysis**: Extract meta tags, headings, and content for SEO auditing
* **Price Monitoring**: Track product prices across e-commerce sites
* **News Monitoring**: Collect latest news and updates from multiple sources
* **Research Automation**: Automatically gather information from academic or research websites

## Error Handling

| Error Type              | Cause                                    | Solution                                                                         |
| ----------------------- | ---------------------------------------- | -------------------------------------------------------------------------------- |
| Invalid URL             | URL is malformed or inaccessible         | Verify URLs are complete and publicly accessible (include http\:// or https\://) |
| Connection Timeout      | Website took too long to respond         | Increase timeout or try again later; check if website is online                  |
| Rate Limit Exceeded     | Too many scraping requests               | Space out requests or upgrade your PixelML/Apify plan                            |
| Bot Detection           | Website blocked the scraping attempt     | Try different crawler\_type or use "playwright:adaptive" for better bot evasion  |
| Content Too Large       | Scraped content exceeds max\_tokens      | Reduce max\_tokens\_per\_url or target specific pages with less content          |
| JavaScript Required     | Page requires JS but using cheerio/jsdom | Switch to playwright:firefox or playwright:chrome for JS execution               |
| Authentication Required | Website requires login                   | This node works with public pages only; use authenticated scraping methods       |

## Notes

* **Crawler Selection**: Use Playwright for JavaScript-heavy sites, Cheerio/JSDOM for faster, static HTML parsing.
* **Token Limits**: Higher max\_tokens increases cost and processing time. Start with lower values and adjust as needed.
* **URL Limits**: Maximum 10 URLs per request. For batch scraping, use multiple node instances or loop structures.
* **Adaptive Mode**: playwright:adaptive intelligently switches between browsers for optimal scraping success.
* **Legal Compliance**: Ensure you have permission to scrape target websites and comply with their robots.txt and terms of service.
* **Performance**: Cheerio is fastest but doesn't execute JavaScript. Playwright is slower but handles dynamic content.
* **Cost Management**: Each scraping operation consumes API credits. Monitor usage especially with high max\_tokens values.
* **Content Format**: Scraped content is returned as plain text. HTML structure is typically stripped out.