Use more info from pydantic models

It is my understanding that at present any descriptive text given within the description of a field within a pydantic model is not considered by the under lying prompt constructors.

class ContentCreatorInfo(BaseModel):
    first_name: Optional[str] = Field(
        ..., description="The first name of the content creator"
    )
    last_name: Optional[str] = Field(
        None, description="The last name of the content creator"
    )
main_topics_covered: Optional[List[str]] = Field(
        None, description="The main topics covered by the content creator"
    )
    bio: Optional[str] = Field(
        None, description="A brief biography of the content creator"
    )

As can be seen above, the descriptive parameter of the Field of the pydantic model can hold a lot of textual information about the fields within a pydantic model.
I propose that such information could be used to create more robust & reliable input/output models for our crews, etc.

Reasoning/Motivation
Having considered the application of AI as both a tech manger & a dev, a common concern was that of repeatability/consistency of output. The ‘generated by AI always check the result’ worries when concidering within commercial environments.
Pydantic models provide a more consistent interface between logical & cognative systems/processes by providing granular control of the cognition used to compile each field value.

Futher Thoughs :bulb:
Assuming that a common task would be to populate all, or part of the custom Pydantic model:
The pydantic-> BasModel-> Field description could be considered as the ‘prompt/cognition’ to use to populate the associated field.

If this is in the wrong category, let me know & I’ll move it

1 Like

crewAI already uses Pydantic to do this

Appologies Matt, but I asked this question in discord & was told that at this time CrewAI does not use the description param of the pydantic Field!

1 Like

Agree,from my implementation, i also found that the framework is not using any description from the model. Even with the instructor package, it’s actually not using it very well as if there is any pydantic error, the code will break. Also, you need to write all the description from the model in the task file, which looks to be duplicates

FYI: I also asked CrewAI GPT Assistant:

But can we trust the answer :slight_smile:

If you watch the context input to the llm either in local logs if you are using a local llm endpoint or say in the agentops logs, you will see the description gets in there. I believe that is the general case. Whether or not that has a profound impact is another question. It probably depends on how big the input context has gotten, the complexity of the info passed, and of course the ability of the model. Thats why if you are using open source models, I would always use a separate function_calling_llm. You can be very specific in the sys prompt to a given instance of the llm. If you are having real problems, break the problem down and give only 1 tool to a given agent and creat a separate instance of the function calling llm with different sys prompt being specific about the tool that agent has been given. Create structures wherever possible. Agentops is very helpful. Langtrace is coming along as well.

1 Like

Keeping this discussion about structured output going: Has anyone used the Response Template (optional) response_template Specifies the response format for the agent. Default is None. of the Agent.
Or,
Converter Class (optional) converter_cls A converter class used to export structured output. Defaults to None. of Task?

To better describe my initial suggestion I’ve created an ‘out of the box’ >> crewai create crew test02 via the cli, then added a couple of minor updates to show a working example.
The LLM used GPT4omini

The crew.py

import json
import os
from typing import Optional, List

from crewai import Agent, Crew, Process, Task
from crewai.project import CrewBase, agent, crew, task
from langchain_openai import ChatOpenAI
from langchain_community.llms.ollama import Ollama
from langchain_groq import ChatGroq
from pydantic import BaseModel, Field


# Uncomment the following line to use an example of a custom tool
# from test02.tools.custom_tool import MyCustomTool

# Check our tools documentations for more information on how to use them
# from crewai_tools import SerperDevTool


@staticmethod
class LLMS:
	def __init__(self):
		self.OpenAIGPT35 = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0.7)
		self.OpenAIGPT4oMini = ChatOpenAI(model_name="gpt-4o-mini", temperature=0.8)
		self.OpenAIGPT4 = ChatOpenAI(model_name="gpt-4", temperature=0.8)
		self.Phi3 = Ollama(model="phi3:mini")
		self.Llama3_1 = Ollama(model="llama3.1")
		# self.Phi3 = Ollama(model="phi3.5:latest")
		# self.Phi3 = ChatOpenAI(model_name="phi3:medium-128k", temperature=0, api_key="ollama", base_url="http://localhost:11434")
		self.groqLama3_8B_3192 = ChatGroq(temperature=0.5, groq_api_key=os.environ.get("GROQ_API_KEY"),
										  model_name="llama3-8b-8192")



class Stage1OutputModel(BaseModel):
    summary: Optional[str] = Field("", description="Explanation of how the report should be read")
    bullet_points: Optional[List[str]] = Field([], description="3 key bullet points from the report")

    def __init__(self):
        super().__init__()

    def get_field_info(self) -> str:
        field_info = "\n"
        for field_name, field_instance in self.model_fields.items():
            field_info += field_name + ", described as: " + field_instance.description + "\n"
        return field_info


@CrewBase
class Test02Crew:
    """Test02 crew"""
    agents_config = 'config/agents.yaml'
    tasks_config = 'config/tasks.yaml'

    def __init__(self):
        self.llms = LLMS()

    @agent
    def researcher(self) -> Agent:
        return Agent(
            config=self.agents_config['researcher'],
            llm=self.llms.OpenAIGPT4oMini,
            # tools=[MyCustomTool()], # Example of custom tool, loaded on the beginning of file
            verbose=True
        )

    @agent
    def reporting_analyst(self) -> Agent:
        return Agent(
            config=self.agents_config['reporting_analyst'],
            llm=self.llms.OpenAIGPT4oMini,
            verbose=True
        )

    @task
    def research_task(self) -> Task:
        model_info = Stage1OutputModel().get_field_info()
        return Task(
            # config=self.tasks_config['research_task'],
            description=" Conduct a thorough research about {topic}   Make sure you find any interesting and relevant information given the current year is 2024.",
            expected_output=f""" Your response must have these exact field names with values as described, : {model_info}""",
            agent=self.researcher()
        )

    @task
    def reporting_task(self) -> Task:
        return Task(
            # config=self.tasks_config['reporting_task'],
            description="""
			Review the summary you got and expand each bullet-point into a full section for a report.
    		Make sure the report is detailed and contains any and all relevant information.
    		""",
            expected_output="""
			 A fully fledge reports with the mains topics, each with a full section of information.
    		Formatted as markdown without '```'
    		""",
            #context=self.research_task(),
            agent=self.reporting_analyst(),
            output_file='report.md'
        )

    @crew
    def crew(self) -> Crew:
        """Creates the Test02 crew"""
        return Crew(
            agents=self.agents,  # Automatically created by the @agent decorator
            tasks=self.tasks,  # Automatically created by the @task decorator
            process=Process.sequential,
            verbose=True,
            memory=True
            # process=Process.hierarchical, # In case you wanna use that instead https://docs.crewai.com/how-to/Hierarchical/
        )

My lack of crew/python knowledge forced me to extract the Task descriptions, expexted_output values from the config files into the Task definition/creation methods of the crew class.!

Stage1OutputModel
A simple Pydantic model with fields that represent ‘typical’ values of the Task.
get_field_info
Returns a prompt ready string describing the model field names and descriptions.

N.B. That the result of get_field_info is used as the main component of the expected_output of the research_task.

Possible Uses

  1. The get_field_info method applied to any Pydantic model allows for a generalized expected_output prompt for a required Pydantic model.
  2. Change the model fields: add, remove & re-run the crew for an updated output from the task.
  3. Change the model field descriptions & get same output structure, but with different content from the Task. Change bullet points from 3 to ?, etc

Example
Add a new model field ‘todo’

class Stage1OutputModel(BaseModel):
	summary: Optional[str] = Field("", description="Explanation of how the report should be read")
	bullet_points: Optional[List[str]] = Field([], description="3 key bullet points from the report")
	todo: Optional[str] = Field("", description="Suggestions for further research")

	def __init__(self):
		super().__init__()

	def get_field_info_json(self) -> str:
		field_info = "\n"
		for field_name, field_instance in self.model_fields.items():
			field_info += field_name + ", described as: " + field_instance.description + "\n"
		return field_info

Above I have added a new field ‘todo’ then ran the same crew again to get the Task output below:

Thought: I now know the final answer  
Final Answer: {
  "summary": "This report synthesizes the latest advancements, applications, and trends in AI Large Language Models (LLMs) for the year 2024. It covers new models, innovative applications across various industries, and predictions for future developments, providing insights essential for stakeholders.",
  "bullet_points": [
    "In 2024, notable models like GPT-4.5 and BERT-Next have been released, showcasing improved contextual understanding and application effectiveness.",
    "AI LLMs are transforming sectors such as healthcare and finance, with applications ranging from diagnostic support to fraud detection, leading to significant operational efficiencies.",
    "Future trends indicate a move towards hyper-personalization, increased regulatory scrutiny, and the emergence of multimodal models that integrate various forms of data."
  ],
  "todo": "Suggestions for further research include exploring the ethical implications of AI LLMs, examining the impact of regulatory frameworks on AI development, and investigating the potential of multimodal models in various applications."
}

fields addressable within next Task
Take note of the mention of both bullet points & summary within the reporting_task description.

Contexts
As can be seen by the commented out context line in the reporting_task, I have tried to get the values via the previous task context, but adding that just broke everything for me!

I hope this better describes how I believe Pydantic Field description params can be better used.

More of a learning exercise, all comments welcome.

Can anyone improve the expected_output prompt template? The challenge issued :muscle:

Example 2
This technique does seem to have a dramatic and direct impact on th eoutput of a Task.

To demonstrate: I have added another new field to the model: geo_regions. The description of geo_regions is the only place where the summary of geo regions is mentioned.

The updated StageOneOutputModel:

class Stage1OutputModel(BaseModel):
	summary: Optional[str] = Field("", description="Explanation of how the report should be read")
	bullet_points: Optional[List[str]] = Field([], description="3 key bullet points from the report")
	todo: Optional[str] = Field("", description="Suggestions for further research")
	geo_regions: Optional[List[str]] = Field([], description="Highlight geographical regions where research is most active")

	def __init__(self):
		super().__init__()

	def get_field_info_json(self) -> str:
		field_info = "\n"
		for field_name, field_instance in self.model_fields.items():
			field_info += field_name + ", described as: " + field_instance.description + "\n"
		return field_info

The updated reporting_task description that references the new model fields todo & geolocations.:

description="""
			Review the summary you got and expand each bullet-point into a full section for a report.
			Make sure the report is detailed and contains any and all relevant information.
			Mention any further research todo and an overview of geographic hot-spots.
			""",

Updating the above gave this output from the research_task:

2024-09-14 18:58:46][DEBUG]: == [AI LLMs Senior Data Researcher
] Task output: {  
    "summary": "This report provides a comprehensive overview of the latest developments in AI Language Learning Models (LLMs) as of 2024. It includes updated statistics, real-world applications, and case studies that demonstrate the practical uses of AI LLMs across various industries. The report also highlights key geographical regions leading in this technology's research and development, along with suggestions for visual elements that could enhance understanding. The information is structured to facilitate easy navigation and comprehension.",  
    "bullet_points": [  
        "Statistics Overview: As of 2024, the global AI LLM market is projected to reach $45 billion, reflecting a growth rate of 25% annually. Over 70% of enterprises report deploying LLMs in at least one of their operations.",  
        "Real-World Applications: Industries such as healthcare, finance, and education are leveraging AI LLMs for tasks ranging from patient interaction and fraud detection to personalized learning experiences, demonstrating their versatility.",  
        "Geographical Insights: North America, particularly the United States, remains the leader in AI LLM research, followed closely by Europe and Asia, with countries like Canada, Germany, and China showing significant advancements and investments in AI technologies."  
    ],  
    "todo": [  
        "Explore case studies from specific industries (e.g., healthcare and finance) to better understand the impact of AI LLMs.",  
        "Research emerging trends in AI ethics related to LLMs, particularly concerning data privacy and bias mitigation.",  
        "Investigate the regulatory landscape in different regions concerning AI LLM deployment."  
    ],  
    "geo_regions": [  
        "North America: The United States is the hub for AI LLM research, with prominent tech companies and universities leading innovations.",  
        "Europe: Countries like Germany and the United Kingdom are making significant contributions, focusing on ethical AI development.",  
        "Asia: China's rapid investment in AI technologies and its integration of LLMs into government services and e-commerce platforms shows a strong commitment to advancing this field."  
    ]  
}

The newly referenced model fields were seen in the final output:

> Finished chain.
 [2024-09-14 18:59:04][DEBUG]: == [AI LLMs Reporting Analyst
] Task output: # Comprehensive Report on AI Language Learning Models (LLMs) as of 2024

This report provides a comprehensive overview of the latest developments in AI Language Learning Models (LLMs) as of 2024. It includes updated statistics, real-world applications, and case studies that demonstrate the practical uses of AI LLMs across various industries. The report also highlights key geographical regions leading in this technology's research and development, along with suggestions for visual elements that could enhance understanding. The information is structured to facilitate easy navigation and comprehension.

## 1. Statistics Overview

As of 2024, the global market for AI Language Learning Models (LLMs) is projected to reach an impressive **$45 billion**, marking a remarkable growth trajectory with an annual increase rate of **25%**. This robust expansion is indicative of the increasing reliance on AI technologies across various sectors. In fact, more than **70%** of enterprises have reported implementing LLMs in at least one operational facet, ranging from customer service automation to content generation. 

This widespread adoption not only underscores the efficiency that LLMs bring to business processes but also highlights a significant shift towards data-driven decision-making and enhanced user experiences. The proliferation of LLMs is transforming workforce dynamics, creating new job roles while also necessitating upskilling and training for existing employees to effectively harness the capabilities of these advanced models.

## 2. Real-World Applications

The versatility of AI LLMs is vividly illustrated across multiple industries, with notable applications in **healthcare**, **finance**, and **education**. 

In **healthcare**, LLMs are revolutionizing patient interaction through chatbots and virtual assistants that provide timely medical advice, schedule appointments, and manage patient records, thereby enhancing patient engagement and operational efficiency. 

The **finance** sector utilizes LLMs for sophisticated fraud detection mechanisms, analyzing transaction patterns, and identifying anomalies that indicate fraudulent activities. 

Furthermore, in the realm of **education**, LLMs enable personalized learning experiences by tailoring educational content to individual student needs and learning paces, thus enhancing educational outcomes and engagement. 

These real-world applications demonstrate how LLMs are not merely technological innovations but critical tools that drive productivity and improve service delivery across diverse fields.

## 3. Geographical Insights

Geographically, **North America**, specifically the **United States**, continues to lead the charge in AI LLM research and development. The U.S. boasts a rich ecosystem of tech companies, startups, and academic institutions that drive innovation in this field. 

Following closely, **Europe** and **Asia** are also emerging as significant players, with countries such as **Canada**, **Germany**, and **China** making substantial investments in AI technologies. In Canada, for instance, the emphasis on ethical AI development complements its advancements in LLM research. Germany's strong industrial base has led to practical applications of LLMs in manufacturing and logistics. Meanwhile, China has rapidly scaled its AI capabilities, focusing on integrating LLMs into its vast data infrastructure. 

The collaborative efforts and competitive spirit in these regions are propelling AI LLMs to the forefront of technological advancement, positioning them as critical drivers of economic growth in the years to come.

## Research Directions

To further enrich this report, it is crucial to explore specific **case studies** from industries like healthcare and finance to elucidate the impact of AI LLMs in real-world scenarios. Such case studies will provide insights into the tangible benefits and challenges faced by organizations during implementation. 

Additionally, probing emerging trends in **AI ethics** related to LLMs is vital, particularly in areas concerning **data privacy** and **bias mitigation**. Understanding how organizations are addressing these ethical challenges will shed light on the responsible deployment of AI technologies. 

Furthermore, investigating the **regulatory landscape** across different regions regarding AI LLM deployment will offer valuable context on the governance frameworks that shape the development and application of these models. 

As we identify geographical hotspots such as North America, Europe, and Asia, detailing their contributions and advances in AI LLMs will provide a comprehensive understanding of the global landscape and future directions in this exciting field. 

In summary, this report aims to present a thorough overview of the state of AI LLMs as of 2024, highlighting the dynamic developments, practical applications, and geographical contributions that define this transformative technology.

**REM
Class fields can be dynamically added, removed & mutated at run time :bulb:

I’m a ‘noob’ at this, there must be a better way of doing the same! The challenge is issued :muscle:

Applied to reporting_task output

class Stage2OutputModel(BaseModel):
	summary: Optional[str] = Field("", description="A describe the reports structure")
	highlights: Optional[List[str]] = Field([], description="each of the expanded bullet list sections")
	further_work: Optional[str] = Field("", description="A summary of any further work identified")
	geo_targeting: Optional[List[str]] = Field([], description="Highlight the best geographic areas to target")
	keywords: Optional[List[str]] = Field([], description="A list of suggested reading to gain a deeper understanding of keywords")
	cohorts: Optional[List[str]] = Field([], description="A summary of typical interest cohorts")

	def __init__(self):
		super().__init__()

	def get_field_info_json(self) -> str:
		field_info = "\n"
		for field_name, field_instance in self.model_fields.items():
			field_info += field_name + ", described as: " + field_instance.description + "\n"
		return field_info

Note the different field_names & descriptions that ‘craft’ the output format.

Apply the same changes that we made to the research_task:

	def reporting_task(self) -> Task:
		model_info = Stage2OutputModel().get_field_info_json()
		return Task(
			# config=self.tasks_config['reporting_task'],
			description="""
			Review the summary you got and expand each bullet-point into a full section for a report.
			Make sure the report is detailed and contains any and all relevant information.
			Mention any further research todo and an overview of geographic hot-spots.
			Provide a short description of any keywords given.
			Show any mention of cohorts in the report.
			""",
			expected_output=f""" Your response must have these exact field names with values as described, : {model_info}""",
			agent=self.reporting_analyst(),
			output_file='report.md'
		)

Mention our new model fields in the description.

Note the exact same expected_output template

now run the crew again

Try changing the initial input topic (main.py)
Output formats appear to be stable :grinning:

My thoughts about Stability
Given that we introduce the model structure in the ‘expected_output’ I’m assuming that a cognitive process is made aware of the model/schema at the point where it considers its response/output. If so, then at this point the context is recent/fresh within the cognitive consideration, but more importantly, more recent than any data/info it has collected during Task execution.

A metaphor to explain:
In which of these situations are you more likely to store the information in the correct format:

  1. I want you to store your response in ‘this format’': Go and do your research, collect your information on this subject and return your result.
  2. Go and do your research, collect your information on this subject and return your result in ‘this format’

I hope we all agree that situation 2 is the most likely to get the correct format of response because in terms of ‘context’ format requirements are the most recent/last consideration.

More Thoughts
If we can have more control over Task output we would have a greater reliability of ‘inputs’ to other Task’s because the output format of the previous Task would be more recent within the next Task input context, consider it as a ‘loose’ inter Task connector/interface. :thinking: Maybe not!, Maybe so! :grin:

As usual, all comment welcome.

Yes. Some of this can be done with Task output structured tools. What might be interesting to tighten the connection between the structured output of task 1 to the input of task 2 is to take the structure definition from task 1 and instruct agent 2 that all the important information it will receive is in that same structure.
Crewai does offer json or pydantic output but as far as I know does not do the same for input although maybe there is a way to do it with Config.

@moto: If you read my ‘More Thoughts’ above, I suggest that because the model structure is added to the context at the point of output consideration by the cognitive process. Technically it’s already fresh within the input context of the next Task (at its inputs)

Or, am I missing something?

Plus: The point of the examples is to show how the description param of a Pydantic field can be used within the context & how this allows for a fixed expected_output template for Tasks handling Pydantic models.

If it’s a useful technique, then I would just like someone more skilled than I in Python & CrewAI to implement a better version of it (hints being thrown here!) :grin:

I do see what youre getting at for sure and I have advocated for structure in dealing with llm’s. My point about the input is if you watch what is actually going in the prompt to the llm you will see that after the end of a task there is all the stuff that crewai adds in the context that will follow your pydantic or json structured output from the last task. Just my opinion but I think it really takes a belt AND suspenders approach to minimize the randomness. I do more science oriented stuff rather than narrative writing sorts of things. My results need to be exact so maybe I over do it.
I liked what you did and will work on trying to use it but add the crewai existing structured output tools to it when I return home.

1 Like