# Why is WebsiteSearchTool not parsing the given URL?

**URL:** <https://community.crewai.com/t/why-is-websitesearchtool-not-parsing-the-given-url/1963>\
**Category:** CrewAI Community Support\
**Tags:** tools\_issues\
**Created:** [December 11, 2024, 4:40pm UTC](https://community.crewai.com/t/why-is-websitesearchtool-not-parsing-the-given-url/1963 "2024-12-11T16:40:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![opahopa](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/opahopa/32/1429_2.png) [@opahopa](https://community.crewai.com/u/opahopa)\
**Post date:** [December 11, 2024, 4:40pm UTC](https://community.crewai.com/t/why-is-websitesearchtool-not-parsing-the-given-url/1963/1 "2024-12-11T16:40:14Z")

</div>

So, i’m trying to build one of my first crews.  
How does the `WebsiteSearchTool` works specifically?

I’m feeding the [Introduction - CrewAI](https://docs.crewai.com/concepts/crew) to an agent through the use of `web_rag_tool = WebsiteSearchTool()` to teach him the template of crewai crews(well, my first idea - is to make the crew which will be making crews for me. possible?😁 ):

```auto
    @agent
    def agent_leader(self):
        return Agent(
            config=self.agents_config['agent_leader'],
            tools=[search_tool, web_rag_tool, file_tool],
            allow_delegation=False,
            llm=self.gpt4o,
            verbose=True,
        )

```

what is see in the **log** :

```auto
# Agent: Lead CrewAI developer
## Thought: In order to make sure our AI agents team is designed optimally, I will first read the CrewAI documentation to understand the framework and check the code samples for creating the crew as outlined in the task instructions. Then, I will break down the project into specific roles for the AI agents, assign tasks, and implement these in the `crew.py` and `main.py` files.
## Using tool: Search in a specific website
## Tool Input: 
"{\"search_query\": \"CrewAI documentation code samples\", \"website\": \"https://docs.crewai.com/concepts/crew\"}"
## Tool Output: 
Relevant Content:
CrewAI home page Search CrewAI docs Search... Navigation Core Concepts Crews Get Started Examples CrewAI home page Community Changelog Get Started Introduction Installation Quickstart Core Concepts Agents Tasks Crews Flows Knowledge LLMs Processes Collaboration Training Memory Planning Testing CLI Tools Using LangChain Tools Using LlamaIndex Tools How to Guides Create Custom Tools Sequential Processes Hierarchical Process Create Your Own Manager Agent Connect to any LLM Customize Agents Coding

projects. After creating your CrewAI project as outlined in the Installation section, you can define your crew in a class that inherits from CrewBase and uses decorators to define agents, tasks, and the crew itself. ​ Example Crew Class with Decorators code from crewai import Agent , Crew , Task , Process from crewai . project import CrewBase , agent , task , crew , before_kickoff , after_kickoff @CrewBase class YourCrewName : """Description of your crew""" # Paths to your YAML configuration

example: Agents and tasks are defined directly within the class without decorators. We manually create and manage the list of agents and tasks. This approach provides more control but can be less maintainable for larger projects. ​ Crew Output The output of a crew in the CrewAI framework is encapsulated within the CrewOutput class.

```

and the **output code** is like:

```auto
# Task 7: Coordination and Communication Agent
class CoordinationAgent(Agent):
    def __init__ (self):
        super(). __init__ ()

    def execute(self):
        self.coordinate_data()

    def coordinate_data(self):
        # Coordinate data between agents (Placeholder function)
        pass

```

So, according to the output the agent didn’t actually parse any code from the page. I tried to run it more than a few times, the model used is `gpt4o` and `gpt4o_mini`

Should i use playwright to properly load the page or am i using the tool wrongly?

Thanks!

---

<div class="post-metadata">

**Author:** ![opahopa](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/opahopa/32/1429_2.png) [@opahopa](https://community.crewai.com/u/opahopa)\
**Post date:** [December 13, 2024, 4:07am UTC](https://community.crewai.com/t/why-is-websitesearchtool-not-parsing-the-given-url/1963/2 "2024-12-13T04:07:16Z")

</div>

> [@opahopa](#):
>
> `https://docs.crewai.com/concepts/crew`

So, i’ve made the tool to use Playwright, tinkered with different tools for almost whole day… only to understand that the issue was that url was [Introduction - CrewAI](https://docs.crewai.com/concepts/crew) instead of [Crews - CrewAI](https://docs.crewai.com/concepts/crews) 🤣 🤣 🤣

---

<div class="post-metadata">

**Author:** ![rokbenko](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/rokbenko/32/492_2.png) [@rokbenko](https://community.crewai.com/u/rokbenko)\
**Post date:** [December 13, 2024, 8:51am UTC](https://community.crewai.com/t/why-is-websitesearchtool-not-parsing-the-given-url/1963/3 "2024-12-13T08:51:14Z")

</div>

> [@opahopa](#):
>
> only to understand that the issue was that url was [Introduction - CrewAI](https://docs.crewai.com/concepts/crew) instead of [Crews - CrewAI](https://docs.crewai.com/concepts/crews) 🤣 🤣 🤣

Happens to the best of us! 🙂

---

<div class="post-metadata">

**Author:** ![rokbenko](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/rokbenko/32/492_2.png) [@rokbenko](https://community.crewai.com/u/rokbenko)\
**Post date:** [December 13, 2024, 8:51am UTC](https://community.crewai.com/t/why-is-websitesearchtool-not-parsing-the-given-url/1963/4 "2024-12-13T08:51:21Z")

</div>