I’ve built a tool which has one string parameter (‘prompt’), and for some reason CrewAI always tries to call it with ‘prompt’ as an object, with ‘description’ string field.
It does error and then call it properly on the retry, but I’d like to fix either my tool or CrewAI so it didn’t need to do this.
What is CrewAI looking for in the tool schema to determine the arguments?
# Agent: Company Research Specialist
## Thought: I will gather comprehensive information about Microsoft, including the current stock price, latest news, emerging trends, and unique insights.
## Using tool: Graphlit RAG prompt tool
## Tool Input:
"{\"prompt\": {\"description\": \"Perform a comprehensive research on the company 'Microsoft'. Gather all necessary information, including the current stock price, latest news, emerging trends, and unique insights.\", \"title\": \"Research on Microsoft\", \"type\": \"string\"}}"
## Tool Output:
I encountered an error while trying to use the tool. This was the error: 1 validation error for PromptInput
prompt
Input should be a valid string [type=string_type, input_value={'description': "Perform ...soft', 'type': 'string'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/string_type.
Tool Graphlit RAG prompt tool accepts these inputs: Accepts user prompt as string.
Prompts LLM with relevant content and returns completion from RAG pipeline. Returns Markdown text from LLM completion.
Uses vector embeddings and similarity search to retrieve relevant content from knowledge base.
Can search through web pages, PDFs, audio transcripts, and other unstructured data..
Moving on then. I MUST either use a tool (use one at time) OR give my best final answer not both at the same time. To Use the following format:
Apologies if I have this wrong, but you are showing an input of:
"{\"prompt\": {\"description\": \"Perform a comprehensive research on
‘prompt’ is a dict {desc: ‘your prompt’ } the error suggests that it’s expecting just a string which is your prompt. " Perform a comprehensive research on the company …"
OK, just re-read your post! give your tool an input type of ‘any’ (input: any) and pull the string prompt from that inside your tool. Difficult to add more without seeing the code (part)
Thanks for your reply @Dabnis. Here’s the definition of the tool, which should help more.
My assumption was that it uses the schema (defined in PromptInput) to know that it just has one callable parameter named ‘prompt’.
What I’m seeing is that CrewAI tries to call the tool with ‘prompt’ as an object with a description field, errors, and then retries, and calls it properly as a single arg.
class PromptInput(BaseModel):
prompt: str = Field(description="Text prompt which is provided to LLM for completion, via RAG pipeline.")
class PromptTool(BaseTool):
name = "Graphlit RAG prompt tool"
description = """Accepts user prompt as string.
Prompts LLM with relevant content and returns completion from RAG pipeline. Returns Markdown text from LLM completion.
Uses vector embeddings and similarity search to retrieve relevant content from knowledge base.
Can search through web pages, PDFs, audio transcripts, and other unstructured data."""
args_schema: Type[BaseModel] = PromptInput
Try setting up the llm specifically and reduce temperature down to 0.1 and play with other parameters. Trying to get accuracy and repetition out of llm ‘s is a difficult proposition when it comes to highly structured things like calling functions.
Totally understand LLMs, as we’re an AI platform vendor.
But I would have thought that CrewAI is responsible for formatting the tools schema into the LLM prompt, and this seemed like something pretty basic that should work out of the box.
It seemed like maybe it wasn’t passing the schema correctly for the tool, is why I’d brought it up.
looking at the context sent to the llm, crewai adds narrative instructions to the llm about formatting the call to the tool but crewai doesn’t actually format anything. its up to the llm. try saying in your system prefix to the llm, "you must use valid json format when calling functions or using tools. It’s hit and miss I’m afraid. lots of little things to do to try and improve it.
It may be true that the LLM isn’t generating the correct tool calling arguments from this, but seems that CrewAI could do a better job with the prompt generation so it works more reliably.
Hard to debug, without being able to see the raw LLM prompts that are generated. Going to see if I can get Langtrace connected to learn more about what’s going on.
Langtrace FTW. CrewAI is formatting the Tool Arguments for the LLM (which is gpt4o-mini by default), and that’s where gpt4o-mini must be getting confused.
UPDATE: After adding explicit usage of gpt-4o, this does work properly. So it was an issue with ‘mini’ not understanding the tool arguments correctly.
The system prompt:
You are Company Research Specialist. As a Company Research Specialist, you excel at synthesizing vast amounts of information into coherent and insightful analyses. Your expertise ensures that all relevant aspects of the company, from financial performance to market positioning, are thoroughly examined.
Your personal goal is: Provide a comprehensive analysis of a specified company, including financial data and strategic insights.
You ONLY have access to the following tools, and should NEVER make up tools that are not listed here:
Tool Name: Graphlit RAG prompt tool
Tool Description: Accepts user prompt as string.
Prompts LLM with relevant content and returns completion from RAG pipeline. Returns Markdown text from LLM completion.
Uses vector embeddings and similarity search to retrieve relevant content from knowledge base.
Can search through web pages, PDFs, audio transcripts, and other unstructured data.
Tool Arguments: {'prompt': {'description': 'Text prompt which is provided to LLM for completion, via RAG pipeline.', 'title': 'Prompt', 'type': 'string'}}
Use the following format:
Thought: you should always think about what to do
Action: the action to take, only one name of [Graphlit RAG prompt tool], just the name, exactly as it's written.
Action Input: the input to the action, just a simple python dictionary, enclosed in curly braces, using \" to wrap keys and values.
Observation: the result of the action
Once all necessary information is gathered:
Thought: I now know the final answer
Final Answer: the final answer to the original input question
So it was llm just not fully understanding the instructions. Sorry if I was not clear above. I knew crewai laid out the format for the llm in its instruction to the llm, but still up to the llm to adhere. One thing I tried was adding my own more detailed instructions in agent descriptions and in the tool description itself, especially working with open source models.