Why is WebsiteSearchTool not parsing the given URL?

CrewAI Community Support
1

So, i’m trying to build one of my first crews.
How does the WebsiteSearchTool works specifically?

I’m feeding the Introduction - CrewAI to an agent through the use of web_rag_tool = WebsiteSearchTool() to teach him the template of crewai crews(well, my first idea - is to make the crew which will be making crews for me. possible?:grin: ):

    @agent
    def agent_leader(self):
        return Agent(
            config=self.agents_config['agent_leader'],
            tools=[search_tool, web_rag_tool, file_tool],
            allow_delegation=False,
            llm=self.gpt4o,
            verbose=True,
        )

what is see in the log:

# Agent: Lead CrewAI developer
## Thought: In order to make sure our AI agents team is designed optimally, I will first read the CrewAI documentation to understand the framework and check the code samples for creating the crew as outlined in the task instructions. Then, I will break down the project into specific roles for the AI agents, assign tasks, and implement these in the `crew.py` and `main.py` files.
## Using tool: Search in a specific website
## Tool Input: 
"{\"search_query\": \"CrewAI documentation code samples\", \"website\": \"https://docs.crewai.com/concepts/crew\"}"
## Tool Output: 
Relevant Content:
CrewAI home page Search CrewAI docs Search... Navigation Core Concepts Crews Get Started Examples CrewAI home page Community Changelog Get Started Introduction Installation Quickstart Core Concepts Agents Tasks Crews Flows Knowledge LLMs Processes Collaboration Training Memory Planning Testing CLI Tools Using LangChain Tools Using LlamaIndex Tools How to Guides Create Custom Tools Sequential Processes Hierarchical Process Create Your Own Manager Agent Connect to any LLM Customize Agents Coding

projects. After creating your CrewAI project as outlined in the Installation section, you can define your crew in a class that inherits from CrewBase and uses decorators to define agents, tasks, and the crew itself. ​ Example Crew Class with Decorators code from crewai import Agent , Crew , Task , Process from crewai . project import CrewBase , agent , task , crew , before_kickoff , after_kickoff @CrewBase class YourCrewName : """Description of your crew""" # Paths to your YAML configuration

example: Agents and tasks are defined directly within the class without decorators. We manually create and manage the list of agents and tasks. This approach provides more control but can be less maintainable for larger projects. ​ Crew Output The output of a crew in the CrewAI framework is encapsulated within the CrewOutput class.

and the output code is like:

# Task 7: Coordination and Communication Agent
class CoordinationAgent(Agent):
    def __init__(self):
        super().__init__()

    def execute(self):
        self.coordinate_data()

    def coordinate_data(self):
        # Coordinate data between agents (Placeholder function)
        pass

So, according to the output the agent didn’t actually parse any code from the page. I tried to run it more than a few times, the model used is gpt4o and gpt4o_mini

Should i use playwright to properly load the page or am i using the tool wrongly?

Thanks!

2

So, i’ve made the tool to use Playwright, tinkered with different tools for almost whole day… only to understand that the issue was that url was Introduction - CrewAI instead of Crews - CrewAI :rofl: :rofl: :rofl:

3

Happens to the best of us! :slightly_smiling_face: