I created an agent and a task, but the task hallucinates, I tried it with more than 1 LLM and the task shows similar hallucination. Here’s the agent and task definition
agents_yaml = {
‘intent_agent’: {
‘role’: ‘Intent Recognition Agent’,
‘goal’: ‘Interpret user queries and select the appropriate domain or workspace for the request.’,
‘backstory’: ‘A skilled language model capable of understanding diverse types of requests and accurately identifying the corresponding database schema.’,
}
Define the agents
intent_agent = Agent(
role=‘Intent Recognition Agent’,
goal=‘Interpret user queries and select the appropriate domain or workspace for the request.’,
verbose=True,
backstory=“A skilled language model capable of understanding diverse types of requests and accurately identifying the corresponding database schema.”,
llm = llm
)
intent_recognition_task = Task(
description=‘Interpret the user's input and determine the relevant domain or workspace.’,
expected_output=‘The name of the domain (e.g., sales, finance, etc.) and any relevant metadata.’,
agent=intent_agent,
).
when I run the crew
result = crew.kickoff(inputs={‘user_query’: ‘Name movie titles released in year 1945. Sort the listing by the descending order of movie’})
print(result)
here’s how the Intent Agent responds:
Agent: Intent Recognition Agent
Task: Interpret the user’s input and determine the relevant domain or workspace.
Agent: Intent Recognition Agent
Final Answer:
The relevant domain for the user’s input is sales. Relevant metadata includes: sales transactions, customer information, product details, sales representatives, sales territories, and historical sales data.
As you can see there’s a complete disconnect between the user query and the intent Agent Output.
I would like Intent Recognition Agent to extract understand users intent from the Natural language Query and handoff to another agent. And by the way this code was also generated using a crewai gpt for given use case. Use case is Narural Language to SQL
You must understand that your user question “‘Name movie titles released in year 1945. Sort the listing by the descending order of movie’.”
Has NO semantic relationship too your Task description: “‘Interpret the user’s input and determine the relevant domain or workspace”, or the expected output "expected_output=‘The name of the domain (e.g., sales, finance, etc.) and any relevant metadata.’, You may as well ask your Task./Agent to tell what colour socks the user is wearing!
I am assuming that English is not your first language.
Let me give you a ‘hyperthetical’ example of what your Task/user question looks/sounds like.
If I said to you, “count the colours in this sentence?”, the sentence being: “What is the distance between the earth and he moon”! It would be ‘impossible’ for you to answer as there’s no ‘semantic’ similarity between what I ask you to do, and what I say!
Q: How do you expect any form of intellegence, artificial or human to aswere the question: "The name of the domain (e.g., sales, finance, etc.) and any relevant metadata… From the user input: “‘Name movie titles released in year 1945. Sort the listing by the descending order of movie” The only option you leave the LLM is to take a guess based on the options and expected output that you have given it.
The best answer I can give you is: You do not understand the basics of LLM’s, LLM prompts, prompt engineering to use CrewAI.
However: CrewAI is a great place to learn these things as it provides the easiest evironment to do so. You can’t ‘copy & paste’ your way through life. Honestly, spend a bit time learning about LLM prompting, the relationship between CrewAI prompts. It has the capacity to change your life
Its working now, there was no problem in the Agent or Task Defintion except I forgot to put parenthesis around user input .thanks for lecturing me but not helping me identify the problem…
Agent: Intent Recognition Agent
Task: Interpret the Get a list of movie titles released in year 1945. Sort the listing by the descending order of movie and determine the relevant domain or workspace.
Agent: Intent Recognition Agent
Final Answer:
The relevant domain for the request to “Get a list of movie titles released in year 1945. Sort the listing by the descending order of movie” is the “Entertainment” domain. The relevant metadata for this domain includes:
Movie Title
Release Year
Genre
Director
Main Cast
Rating
This schema will allow you to retrieve and organize the list of movie titles from the year 1945 in descending order.
Agent: Table Selector Agent
Task: Based on the user’s intent, identify the relevant tables from the database schema. Provide the user an opportunity to review and adjust table selection.
Agent: Table Selector Agent
Final Answer:
Relevant Tables:
Movies
Where Clause from the database schema:
SELECT Movie_Title
FROM Movies
WHERE Release_Year = 1945
ORDER BY Movie_Title DESC;
Yes you figured it out. I faced the same think. For me Crew absolutely took over and started generating its own input ignoring what i have give in the kickoff function. Finally i figured out that what ever you have given in the kickoff function
{“inputparam”:“values”} must be embedded into your agent or task goal or description. What surprised me is this not clear in any of the documentations. I would request crewAI team to document these things upfront.
While I am pleased that you got it working, I still stand by my comments based on what you ‘disclosed’. I still recommend that you take some time and view the free course that I mention above.
If you had declared that your crew input was a DB query my responses would have been different.
I did miss the ‘missing opening curly brace’ , however the error was not related to your question of the Agent hallucinating.