PydanticUserError: FirecrawlCrawlWebsiteTool is not fully defined; you should define FirecrawlApp

General
1

I am trying to use the FirecrawlCrawlWebsiteTool as described here but when running this code:

from crewai_tools import FirecrawlCrawlWebsiteTool
tool = FirecrawlCrawlWebsiteTool(url=‘https://www.firecrawl.dev’)
text = tool.run()
print(text)

…I am getting this error:

pydantic.errors.PydanticUserError: FirecrawlCrawlWebsiteTool is not fully defined; you should define FirecrawlApp, then call FirecrawlCrawlWebsiteTool.model_rebuild().

2

@Ben Try to create a custom tool for Firecrawl.

First, install Firecrawl SDK:

pip install firecrawl

Second, create a custom tool for Firecrawl:

tools.py

from firecrawl.firecrawl import FirecrawlApp

class MyCustomTools:
    def _firecrawl():
        """
        Returns an instance of Firecrawl with the given API key.
        """

        return FirecrawlApp(api_key="xxxxx")

    @tool("Firecrawl crawl custom tool")
    def firecrawl_crawl_tool(url: str) -> str:
        """
        Crawls the given URL.
        """

        response = MyCustomTools._firecrawl().crawl_url(url)

        return response

    @staticmethod
    def get_all_firecrawl_tools():
        """
        Returns all Firecrawl tools available.
        """

        return [MyCustomTools.firecrawl_crawl_tool]

Third, use the custom tool:

main.py

from crewai import Agent
from tools import MyCustomTools

firecrawl_tools = MyCustomTools().get_all_firecrawl_tools()

my_agent = Agent(
    ...,
    tools=firecrawl_tools,
)
1 Like
7

Just wanted to say that I didn’t manage to get your example working. Maybe I made a mistake. Then I took the CustomTool example from the CrewAI website and adapted it based on the additional logic I found from the SerperDevTool. Below is the code, and it works now. Many thanks to @rokbenko for your suggestion which helped me to look in the right direction! :slight_smile:


import os
from typing import Type, Any
from crewai.tools import BaseTool
from pydantic import BaseModel, Field
from firecrawl import FirecrawlApp

class MyToolInput(BaseModel):
    """Input schema for MyCustomTool."""
    url: str = Field(..., description="URL of the website to scrape.")

class MyCustomTool(BaseTool):
    name: str = "Firecrawl Scrape Website Tool"
    description: str = "This tool scrapes a website and returns the text content."
    args_schema: Type[BaseModel] = MyToolInput

    def _run(self, **kwargs: Any) -> Any:

        url = kwargs.get("url")
        app = FirecrawlApp(api_key=os.getenv('FIRECRAWL_API_KEY'))
        scrape_result = app.scrape_url(url, params={'formats': ['markdown', 'html']})

        return scrape_result


tool = MyCustomTool()
text = tool.run(url='https://en.wikipedia.org/wiki/Artificial_intelligence')
print(text['markdown'])
1 Like