I am trying to use the FirecrawlCrawlWebsiteTool as described here but when running this code:
from crewai_tools import FirecrawlCrawlWebsiteTool
tool = FirecrawlCrawlWebsiteTool(url=‘https://www.firecrawl.dev’)
text = tool.run()
print(text)
…I am getting this error:
pydantic.errors.PydanticUserError: FirecrawlCrawlWebsiteTool is not fully defined; you should define FirecrawlApp, then call FirecrawlCrawlWebsiteTool.model_rebuild().
from firecrawl.firecrawl import FirecrawlApp
class MyCustomTools:
def _firecrawl():
"""
Returns an instance of Firecrawl with the given API key.
"""
return FirecrawlApp(api_key="xxxxx")
@tool("Firecrawl crawl custom tool")
def firecrawl_crawl_tool(url: str) -> str:
"""
Crawls the given URL.
"""
response = MyCustomTools._firecrawl().crawl_url(url)
return response
@staticmethod
def get_all_firecrawl_tools():
"""
Returns all Firecrawl tools available.
"""
return [MyCustomTools.firecrawl_crawl_tool]
Third, use the custom tool:
main.py
from crewai import Agent
from tools import MyCustomTools
firecrawl_tools = MyCustomTools().get_all_firecrawl_tools()
my_agent = Agent(
...,
tools=firecrawl_tools,
)
Just wanted to say that I didn’t manage to get your example working. Maybe I made a mistake. Then I took the CustomTool example from the CrewAI website and adapted it based on the additional logic I found from the SerperDevTool. Below is the code, and it works now. Many thanks to @rokbenko for your suggestion which helped me to look in the right direction!
import os
from typing import Type, Any
from crewai.tools import BaseTool
from pydantic import BaseModel, Field
from firecrawl import FirecrawlApp
class MyToolInput(BaseModel):
"""Input schema for MyCustomTool."""
url: str = Field(..., description="URL of the website to scrape.")
class MyCustomTool(BaseTool):
name: str = "Firecrawl Scrape Website Tool"
description: str = "This tool scrapes a website and returns the text content."
args_schema: Type[BaseModel] = MyToolInput
def _run(self, **kwargs: Any) -> Any:
url = kwargs.get("url")
app = FirecrawlApp(api_key=os.getenv('FIRECRAWL_API_KEY'))
scrape_result = app.scrape_url(url, params={'formats': ['markdown', 'html']})
return scrape_result
tool = MyCustomTool()
text = tool.run(url='https://en.wikipedia.org/wiki/Artificial_intelligence')
print(text['markdown'])