I have a Searcher, with a SearchTask using the SerperDevTool & a Scraper agent with a ScrapeTask.
In the tasks.yaml file, I include in the ScrapeTask: Scrape ALL of the websites returned from the SearchTask. Nothing comes through to this task. What appears on screen is something about example.com.
When I include in the ScrapeTask: context=[self.SearchTask] I get an error:
AttributeError: ‘function’ object has no attribute ‘get’.
Anybody have a suggestion?
@Baruch_Bashan Can you please edit the question and add the code?
Hello Rok! Thank you for writing and for inviting me to share my code. I’m still unsure how much is too much code to share.
I must adjust my question: It appears that the results from the first agent (Searcher) is getting through to the second agent (Scraper).
By implementing step_callback calls for my Searcher and Scraper agents, I am able to save the SearchTask output (tools=[SerperDevTool( api_key=srprKey, n_results=numRet )], and the results from the ScrapeTask.
I see that the SearchTask returns the correct number of results, with the title, link, and snippet. In my case numRet == 100.
For each task tied to each agent, I have a output_file parameter.
What I’m seeing is that the output for the SearchTask returns only 4 results, with none of them having the required title, link, and snippet.
Worse yet, the output for the ScrapeTask is for only 1 of those results.
Below is my code:
agents.yaml
Searcher:
role: >
Senior Web Searcher
goal: >
Search for and identify funding sources focused on projects that might fund this topic: {topic} and {phrase}.
backstory: >
An expert researcher with vast experience in finding grant and funding organizations.
Known for your ability to find the most relevant information and present it in a clear and concise manner.
Scraper:
role: >
Senior Web Scraper
goal: >
Scrape the ALL of the websites returned from the Searcher Agent and look for sentences and links that include any of the following words: {keywords}.
backstory: >
Skilled in scraping all types of websites to gain the required information.
tasks.yaml
SearchTask:
description: >
Perform a web search about {topic} and {phrase} to find relevant URLs.
expected_output: >
A detailed report of ALL URLs related to {topic} and {phrase}, including the organization name, website link, contact link, and all sentences and links that include any of the following words: {keywords}.
If you get any error when trying to reach the website, DO NOT try again. Skip this website url and go on to the next website url.
agent: Searcher
ScrapeTask:
description: >
Scrape ALL of the website URLs returned from the SearchTask. If you get any error when trying to reach the website, DO NOT try again. Skip this website url and go on to the next website url.
Use the URLs provided by the SearchTask to gather sentences and links in each website that include any of the following words: {keywords}.
Make sure you find all potential organizations with available funding for 2024 and later.
expected_output: >
Detailed information extracted from All website URLs provided by SearchTask, including the organization name, website link, contact link, and all required sentences and links.
agent: Scraper
@CrewBase
class Myt6Crew:
“”“Myt6 crew”“”
@agent
def Searcher(self) → Agent:
return Agent(
config=self.agents_config[“Searcher”],
tools=[searchTool], # searchTool internetSearchTool search_tool , scrapeTool
llm=llmModel,
function_calling_llm=llmModel,
max_rpm=100,
max_iter=1,
step_callback=SearchTaskCallbackFunction,
verbose=True
)
@agent
def Scraper(self) → Agent:
return Agent(
config=self.agents_config[“Scraper”],
tools=[scrapeTool], # wbSrchTool
llm=llmModel,
max_iter=1,
step_callback=ScrapeTaskCallbackFunction,
verbose=True
)
@task
def SearchTask(self) → Task:
return Task(
config=self.tasks_config[“SearchTask”],
output_file=“Searched.txt”,
)
@task
def ScrapeTask(self) → Task:
return Task(
config=self.tasks_config[“ScrapeTask”],
#context=[self.SearchTask],
output_file=“Scraped.txt”,
)
@crew
def crew(self) → Crew:
“”“Creates the Myt6 crew”“”
return Crew(
agents=self.agents,
tasks=self.tasks,
process=Process.sequential,
verbose=True
)
? Why does an agent who’s only task is to use the SerperDevTool with a query such as below, need an LLM? Is this LLM (Ollama mistral) causing the Searcher Agent and its SearchTask to only report on 4 of 100 search hits, and without the requested information? As you can see in the yaml files, I specifically tell them to deal with all results.
Thank you in advance for any suggestions you may have.
Baruch