Why would I want to do this?
Summary: A crew that tunes its Agent prompts to be more efficient at hydrating a known Pydantic model.
As some will know, I’m presently looking at ways of using more info from a Pydantic model within crew’s, specifically the description parameter of the Field. A topic on the discussion can be found here
As a purely academic exercise I contemplated the possibility of getting an Task/Agent to analyze the field descriptions of the Pydantic model and assess how efficient the existing crew Agent prompts are in respect of collecting information required to hydrate the model.
The process generates suggestions for improved: role;goal;backstory prompts for all crew Agents.
I created a tool that accepts a crew instance as a parameter, allowing for code within the tool to update Agent values within the crew.
The code below shows that this is a far as I have got! I have the question: What to change?
Here’s the complete crew.py
import os
from typing import Optional, List, Any
from crewai import Agent, Crew, Process, Task
from crewai.project import CrewBase, agent, crew, task
from decorator import append
from langchain_openai import ChatOpenAI
from langchain_community.llms.ollama import Ollama
from langchain_groq import ChatGroq
from pydantic import BaseModel, Field
# Tools
from crewai_tools import SerperDevTool
from crews.pydantic.src.pydantic.tools.agent_pmt_updater import AgentPromptUpdater
from crews.pydantic.src.pydantic.tools.pyd_mdl import PydanticModelActions
@staticmethod
class LLMS:
def __init__(self):
self.OpenAIGPT35 = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0.7)
self.OpenAIGPT4oMini = ChatOpenAI(model_name="gpt-4o-mini", temperature=0.8)
self.OpenAIGPT4o = ChatOpenAI(model_name="gpt-4o", temperature=0.8)
self.OpenAIGPT4 = ChatOpenAI(model_name="gpt-4", temperature=0.8)
# self.Phi3 = Ollama(model="phi3:mini")
self.Llama3_1 = Ollama(model="llama3.1")
self.Phi3 = Ollama(model="phi3:medium-128k")
# self.Phi3 = ChatOpenAI(model_name="phi3:medium-128k", temperature=0, api_key="ollama", base_url="http://localhost:11434")
self.groqLama3_8B_3192 = ChatGroq(temperature=0.5, groq_api_key=os.environ.get("GROQ_API_KEY"),
model_name="llama3-8b-8192")
class DabnisBase(BaseModel):
def __init__(self, /, **data: Any):
super().__init__(**data)
def get_field_info(self) -> str:
field_info = ""
for field_name, field_instance in self.model_fields.items():
# field_type=str(field_instance.annotation)
field_info += field_name + ", type " + str(field_instance.annotation) + " described as: " + str(
field_instance.description) + ". "
return field_info
# Pydantic Model
class TopicResearchModel(DabnisBase):
topic: Optional[str] = Field("", description="The report topic.")
report: Optional[str] = Field("", description="The detailed report.")
guidance: Optional[str] = Field("", description="Guidance on how to read the resulting report")
summary: Optional[str] = Field("", description="A concise summary of the report")
bullet_points: Optional[List[str]] = Field([], description="5 key bullet points from the report")
todo: Optional[str] = Field("", description="Suggestions for further research")
geo_regions: Optional[List[str]] = Field([],
description="Highlight geographical regions where research is most active")
keywords: Optional[List[str]] = Field([],
description="A list of at most 10 topic related keywords that should be understood to best ingest the report contents")
cohorts: Optional[List[str]] = Field([], description="A list of at most 10 cohorts where research is most active")
def __init__(self, /, **data: Any):
super().__init__(**data)
# Check our tools documentations for more information on how to use them
# from crewai_tools import SerperDevTool
@CrewBase
class PydanticCrew:
"""Pydantic crew"""
agents_config = 'config/agents.yaml'
tasks_config = 'config/tasks.yaml'
def __init__(self, pd_mdl: Any):
super().__init__()
self.llms = LLMS()
self.pyd_model = pd_mdl
self.OnlineSearchTool = SerperDevTool()
def _agents(self) -> str:
agts = "\n"
# Iterate over agents_config and append formatted agent details
for agent_name, agent_instance in self.agents_config.items():
# Create a string from agent details
details = "\n"
for key, value in agent_instance.items():
details += str(key) + ":" + str(value)
# print(f"key: {key}, value: {value}")
# print(f"details: {details} \n")
agts += f"{agent_name} \n{details}"
# print(f"agts: {agts}")
# Add the agent name and details to the list
# agts.append(f"{agent_name} \n{details}")
# Now concatenate all agents in agts with double newline between them
# response = "\n\n".join(agts)
return agts
@agent
def researcher(self) -> Agent:
return Agent(
role="Senior Online Researcher",
goal="""
To discover and validate the most up-to-date and reliable online sources, academic papers, and industry reports
for the task that I am given.
""",
backstory="""
A self-evolving, AI-enhanced researcher with a passion for discovery.
I have evolved into a fully autonomous entity, capable of conducting deep-dive
research into complex, interdisciplinary fields.
I ensure the integration of the latest and most credible information into the research process.
""",
# tools=[self.OnlineSearchTool], # Example of custom tool, loaded on the beginning of file
verbose=True,
llm=self.llms.OpenAIGPT4oMini
)
@agent
def model_hydration_planner(self) -> Agent:
return Agent(
role="Pydantic Model Hydration Planner",
goal="""
To meticulously examine Pydantic model fields and create a detailed plan
to follow that will provide accurate and efficient hydration of the model field values.
""",
backstory=
"You excel at interpreting Pydantic models structures and field descriptions to "
"gain insight into the value needed for each field. "
"As an expert project planner you use these insights to create a detailed ordered plan to "
"hydrate the model field values with precision.",
# tools=[self.OnlineSearchTool], # Example of custom tool, loaded on the beginning of file
verbose=True,
allow_delegation=False,
llm=self.llms.OpenAIGPT4oMini
)
@agent
def model_prompt_tuning(self) -> Agent:
return Agent(
role="Pydantic Prompt Optimization Expert",
goal="""
To produce prompts that are optimized for a Pydantic model field requirements.
""",
backstory=
"You are an expert AI prompt developer "
"Your deep understanding of the Agent context makes you an expert at this such tasks .",
# tools=[self.OnlineSearchTool], # Example of custom tool, loaded on the beginning of file
verbose=True,
allow_delegation=False,
llm=self.llms.OpenAIGPT4oMini
)
# @agent
# def model_hydrator(self) -> Agent:
# return Agent(
# role="Pydantic Model Hydration Planner",
# goal="""
# To meticulously examine Pydantic model fields and create a detailed plan that guides crew member agents
# in the accurate and efficient hydration of model field values.
# """,
# backstory="""
# You excel at interpreting Pydantic models and field descriptions to
# gain insight into the exact value needed for each field.
# As an expert you use these insights to create a detailed plan to hydrate the model field values for crew member agents to follow
# to hydrate fields with precision.
# """,
# # tools=[self.OnlineSearchTool], # Example of custom tool, loaded on the beginning of file
# verbose=True,
# allow_delegation=True,
# llm=self.llms.Llama3_1
# )
@task
def model_prompt_tuning_task(self) -> Task:
mdl = TopicResearchModel()
model_fields = mdl.get_field_info()
agents = self._agents()
# pyd_mdl_actions = PydanticModelActions(
# pyd_mdl=self.pyd_model
# )
prompt_updater=AgentPromptUpdater(crew=self)
return Task(
description=
" By considering these Pydantic model fields:" + model_fields + " that the crew is tasked with hydrating, "
"update the Agent prompts with any improvements and optimizations based on the field descriptions to : role; goal; backstory prompts of these "
"existing crew member:" + str( agents) + " that will provide the most accurate and efficient hydration "
"of the model field values.",
expected_output=
"""
A report of each prompt showing present and suggested improved prompt along with an explanation of why it is better
HIGHLIGHTING anything related to the Pydantic model fields descriptions
""",
tools=[prompt_updater],
agent=self.model_prompt_tuning(),
output_file='agent_prompt_updates.md'
)
@task
def model_hydration_plan_task(self) -> Task:
mdl = TopicResearchModel()
model_fields = mdl.get_field_info()
agents = self._agents()
return Task(
description=
"Considering these Pydantic model fields:" + model_fields + " Using the topic value of: ({topic}), "
"create a detailed plan for crew member agents to hydrate the remaining field values, "
"It's IMPORTANT that the plan considers the interdependency of field values and the order "
"of field hydration, at the end of the plan ALL fields should have been hydrated in the correct order. "
"The plan should provide field value validations "
"Analise and suggest improvements to existing crew member:" + str( agents) + " text prompts. "
"If the existing crew could be improved to produce 'best in class' with new members, suggest with duties. "
"DO NOT action any part of the plan.",
expected_output="""
A detailed ordered plan of crew member tasks each task having a format of: Task; Key Considerations, Crew Member,
for crew member agents to hydrate the values of the Pydantic model.
List ALL used crew member agents marked as 'existing', or 'suggested' at end of plan with a description of the Crew Members duties plus a count of assigned tasks""",
agent=self.model_hydration_planner(),
output_file='hydration_plan.md'
)
@crew
def crew(self) -> Crew:
"""Creates the Pydantic crew"""
return Crew(
agents=self.agents, # Automatically created by the @agent decorator
tasks=self.tasks, # Automatically created by the @task decorator
process=Process.sequential,
verbose=True,
# memory=True,
# planning=True,
manager_llm=self.llms.OpenAIGPT4oMini
# process=Process.hierarchical, # In case you wanna use that instead https://docs.crewai.com/how-to/Hierarchical/
)
This is the tool:
from typing import Any, Optional, Type
from crewai_tools import BaseTool
from pydantic import BaseModel, Field
class AgentPromptInput(BaseModel):
agent_name: Optional[str] = Field(None, description='The name Agent that the prompt belongs to.')
prompt_name: Optional[str] = Field("get", description='The prompt name, derived directly, or via interpretation of a synonyms ')
prompt_value: str = Field(..., description='The text that will be stored in the prompt ')
class AgentPromptUpdater(BaseTool):
name: str = "Agent Prompt Updater"
description: str = "Updates agent prompts."
args_schema: Type[BaseModel] = AgentPromptInput
# return_schema: Type[BaseModel] = WebSearchOutput
attached_crew: Any = None
def __init__(self, crew: Any):
super().__init__()
self.attached_crew = crew
def _run(self, agent_name: str, prompt_name: str, prompt_value: str ) -> str:
print(f"agent_name:{agent_name}")
print(f"prompt_name:{prompt_name}")
print(f"prompt_value:{prompt_value}")
# TODO : Implement Agent prompt updater
return f"prompt {prompt_name} of Agent {agent_name} updated to: {prompt_value}"
Here are some screen-shots of what is seen during debugging the agent_prompt_updater tool.
I can see my three agents, however when I expand one of them I’m confused of where I could update an Agents prompts.
For completeness here is the output of the 'model_prompt_tuning_task:
hought: I will analyze the current prompts for the crew member roles of "researcher" and "model_hydration_planner," and see how they can be optimized based on the Pydantic model fields descriptions provided. I will then update the prompts accordingly to ensure they align better with the required model fields.
**Current Prompts:**
1. **Researcher**
- **Role**: AI LLM Agents Senior Data Researcher
- **Goal**: Uncover cutting-edge developments in AI LLM Agents
- **Backstory**: You're a seasoned researcher with a knack for uncovering the latest developments in AI LLM Agents. Known for your ability to find the most relevant information and present it in a clear and concise manner.
2. **Model Hydration Planner**
- **Role**: AI LLM Agents Reporting Analyst
- **Goal**: Create detailed reports based on AI LLM Agents data analysis and research findings
- **Backstory**: You're a meticulous analyst with a keen eye for detail. You're known for your ability to turn complex data into clear and concise reports, making it easy for others to understand and act on the information you provide.
**Suggested Improvements:**
### Researcher Prompt Improvements
- **Present Prompt**:
- **Role**: AI LLM Agents Senior Data Researcher
- **Goal**: Uncover cutting-edge developments in AI LLM Agents
- **Backstory**: You're a seasoned researcher with a knack for uncovering the latest developments in AI LLM Agents. Known for your ability to find the most relevant information and present it in a clear and concise manner.
- **Suggested Prompt**:
- **Role**: AI LLM Agents Senior Data Researcher
- **Goal**: Identify and summarize emerging trends and key insights in AI LLM Agents research, including geographical regions and active cohorts.
- **Backstory**: You're a seasoned researcher with expertise in AI LLM Agents. You excel at identifying relevant information, crafting concise reports that encompass essential topics, geographical highlights, and key cohorts, ensuring clarity and accessibility for the audience.
**Explanation**: The suggested prompt emphasizes the need to include geographical regions and cohorts, aligning with the Pydantic model fields. This makes the research goals more comprehensive and relevant.
### Model Hydration Planner Prompt Improvements
- **Present Prompt**:
- **Role**: AI LLM Agents Reporting Analyst
- **Goal**: Create detailed reports based on AI LLM Agents data analysis and research findings
- **Backstory**: You're a meticulous analyst with a keen eye for detail. You're known for your ability to turn complex data into clear and concise reports, making it easy for others to understand and act on the information you provide.
- **Suggested Prompt**:
- **Role**: AI LLM Agents Reporting Analyst
- **Goal**: Produce comprehensive reports that incorporate detailed analysis, concise summaries, and actionable bullet points regarding AI LLM Agents, including suggestions for further research and relevant keywords.
- **Backstory**: You are a meticulous analyst with expertise in converting complex AI LLM Agents data into structured reports. You focus on delivering clear summaries, highlighting geographical regions, and identifying key keywords and cohorts to provide actionable insights for ongoing research.
**Explanation**: The suggested prompt incorporates specific elements from the Pydantic model fields, such as detailed analysis, summaries, bullet points, suggestions for further research, and keywords. This makes the goal more aligned with the expected data structure and reporting requirements.
Now, I will update the Agent prompts using the `Agent Prompt Updater` tool.
### Updating Prompts
**1. Researcher Prompt Update**
Thought: I will update the researcher prompt to reflect the improvements.
Action: Agent Prompt Updater
Action Input: {"agent_name": "researcher", "prompt_name": "goal", "prompt_value": "Identify and summarize emerging trends and key insights in AI LLM Agents research, including geographical regions and active cohorts."}
The ‘Explanations’ above give me hope 
However I can not see how to access an Agent by ‘name’.
Any comments/suggestions welcome.
P.S. Yes I can see a natural progression to researching dynamic Agent/Task creation and crews creating crews!! **REM ALL purely academic research/learning.