---
title: "Spider Agent"
description: "Web scraping tools for AI agents."
icon: "spider"
---

<Note>
  **Prerequisites**
  - Python 3.10 or higher
  - PraisonAI Agents package installed
  - `scrapy` package installed
</Note>

## Spider Tools

Use Spider Tools to crawl and scrape web content with AI agents.

<Steps>
  <Step title="Install Dependencies">
    First, install the required packages:
    ```bash
    pip install praisonaiagents scrapy
    ```
  </Step>

  <Step title="Import Components">
    Import the necessary components:
    ```python
    from praisonaiagents import Agent, Task, PraisonAIAgents
    from praisonaiagents.tools import scrape_page, extract_links, crawl, extract_text
    ```
  </Step>

  <Step title="Create Agent">
    Create a web scraping agent:
    ```python
    spider_agent = Agent(
        name="WebSpider",
        role="Web Scraping Specialist",
        goal="Extract and analyze web content efficiently.",
        backstory="Expert in web scraping and content extraction.",
        tools=[scrape_page, extract_links, crawl, extract_text],
        self_reflect=False
    )
    ```
  </Step>

  <Step title="Define Task">
    Define the scraping task:
    ```python
    scraping_task = Task(
        description="Scrape product information from an e-commerce website.",
        expected_output="Structured product data with prices and descriptions.",
        agent=spider_agent,
        name="product_scraping"
    )
    ```
  </Step>

  <Step title="Run Agent">
    Initialize and run the agent:
    ```python
    agents = PraisonAIAgents(
        agents=[spider_agent],
        tasks=[scraping_task],
        process="sequential"
    )
    agents.start()
    ```
  </Step>
</Steps>

## Understanding Spider Tools

<Card title="What are Spider Tools?" icon="question">
  Spider Tools provide web scraping capabilities for AI agents:
  - Page scraping and downloading
  - Content extraction and filtering
  - Link discovery and crawling
  - HTML parsing and cleaning
  - Data structuring and formatting
</Card>

## Key Components

<CardGroup cols={2}>
  <Card title="Spider Agent" icon="user-robot">
    Create specialized scraping agents:
    ```python
    Agent(tools=[scrape_page, extract_content, crawl_links, parse_html, structure_data])
    ```
  </Card>
  <Card title="Spider Task" icon="list-check">
    Define scraping tasks:
    ```python
    Task(description="scraping_query")
    ```
  </Card>
  <Card title="Process Types" icon="arrows-split-up-and-left">
    Sequential or parallel processing:
    ```python
    process="sequential"
    ```
  </Card>
  <Card title="Scraping Options" icon="sliders">
    Customize scraping parameters:
    ```python
    max_depth=2, delay=1
    ```
  </Card>
</CardGroup>

## Examples

### Basic Web Scraping Agent

```python
from praisonaiagents import Agent, Task, PraisonAIAgents
from praisonaiagents.tools import scrape_page, extract_links, crawl, extract_text

# Create search agent
agent = Agent(
    name="WebCrawler",
    role="Web Scraping Specialist",
    goal="Extract and analyze web content efficiently.",
    backstory="Expert in web scraping and content extraction.",
    tools=[scrape_page, extract_links, crawl, extract_text],
    self_reflect=False
)

# Define task
task = Task(
    description="Scrape and analyze the content from 'https://example.com'",
    expected_output="Extracted content with links and text analysis",
    agent=agent,
    name="web_scraping"
)

# Run agent
agents = PraisonAIAgents(
    agents=[agent],
    tasks=[task],
    process="sequential"
)
agents.start()
```

### Advanced Scraping with Multiple Agents

```python
# Create scraping agent
scraper_agent = Agent(
    name="Scraper",
    role="Content Scraper",
    goal="Extract web content systematically.",
    tools=[scrape_page, crawl_links, parse_html],
    self_reflect=False
)

# Create analysis agent
analysis_agent = Agent(
    name="Analyzer",
    role="Content Analyst",
    goal="Analyze and structure scraped content.",
    backstory="Expert in data analysis and organization.",
    tools=[extract_content, structure_data],
    self_reflect=False
)

# Define tasks
scraping_task = Task(
    description="Scrape product data from multiple pages.",
    agent=scraper_agent,
    name="product_scraping"
)

analysis_task = Task(
    description="Analyze and structure the scraped product data.",
    agent=analysis_agent,
    name="data_analysis"
)

# Run agents
agents = PraisonAIAgents(
    agents=[scraper_agent, analysis_agent],
    tasks=[scraping_task, analysis_task],
    process="sequential"
)
agents.start()
```

## Best Practices

<AccordionGroup>
  <Accordion title="Agent Configuration">
    Configure agents with clear scraping focus:
    ```python
    Agent(
        name="WebScraper",
        role="Content Extraction Specialist",
        goal="Extract web content ethically and efficiently",
        tools=[scrape_page, extract_content, crawl_links, parse_html, structure_data]
    )
    ```
  </Accordion>

  <Accordion title="Task Definition">
    Define specific scraping objectives:
    ```python
    Task(
        description="Extract product prices and descriptions from e-commerce site",
        expected_output="Structured product database"
    )
    ```
  </Accordion>
</AccordionGroup>

## Common Patterns

### Web Scraping Pipeline
```python
# Scraping agent
scraper = Agent(
    name="Scraper",
    role="Web Scraper",
    tools=[scrape_page, crawl_links, parse_html]
)

# Processing agent
processor = Agent(
    name="Processor",
    role="Data Processor",
    tools=[extract_content, structure_data]
)

# Define tasks
scrape_task = Task(
    description="Scrape website content",
    agent=scraper
)

process_task = Task(
    description="Process scraped content",
    agent=processor
)

# Run workflow
agents = PraisonAIAgents(
    agents=[scraper, processor],
    tasks=[scrape_task, process_task]
)