Agent Hallucination

I created an agent and a task, but the task hallucinates, I tried it with more than 1 LLM and the task shows similar hallucination. Here’s the agent and task definition

agents_yaml = {
‘intent_agent’: {
‘role’: ‘Intent Recognition Agent’,
‘goal’: ‘Interpret user queries and select the appropriate domain or workspace for the request.’,
‘backstory’: ‘A skilled language model capable of understanding diverse types of requests and accurately identifying the corresponding database schema.’,
}

Define the agents

intent_agent = Agent(
role=‘Intent Recognition Agent’,
goal=‘Interpret user queries and select the appropriate domain or workspace for the request.’,
verbose=True,
backstory=“A skilled language model capable of understanding diverse types of requests and accurately identifying the corresponding database schema.”,
llm = llm
)

Hi @Gyanesh_Kumar,
Agents/Tasks do not hallucinate, LLM’s do. Have you tried playing around with your prompt?

Also: You do not give the contents of your Task.
Without the above it would be difficult to provide suggestions.

Personally I would paste your Agent/Task into GPT/Gemini, etc. Explain the issue & see what you get back.

Hope you get it sorted :hand_with_index_finger_and_thumb_crossed:

here’s the task details:

Define the tasks

intent_recognition_task = Task(
description=‘Interpret the user's input and determine the relevant domain or workspace.’,
expected_output=‘The name of the domain (e.g., sales, finance, etc.) and any relevant metadata.’,
agent=intent_agent,
).

when I run the crew

result = crew.kickoff(inputs={‘user_query’: ‘Name movie titles released in year 1945. Sort the listing by the descending order of movie’})
print(result)

here’s how the Intent Agent responds:

Agent: Intent Recognition Agent

Task: Interpret the user’s input and determine the relevant domain or workspace.

Agent: Intent Recognition Agent

Final Answer:

The relevant domain for the user’s input is sales. Relevant metadata includes: sales transactions, customer information, product details, sales representatives, sales territories, and historical sales data.

As you can see there’s a complete disconnect between the user query and the intent Agent Output.

I can see absolutely no connection between your task and the semantics of your user input!

Your task is to identify the ‘relevant domain’ within the users input.

Then ayou ask it to: ‘‘Name movie titles released in year 1945. Sort the listing by the descending order of movie’

What has the contents/nature of the question have to do with anything that you have asked your task to consider?

TBH: It did well to get to ‘sales’

Both totally different contexts, unrelated!
TBH: It does not make sense.

I would like Intent Recognition Agent to extract understand users intent from the Natural language Query and handoff to another agent. And by the way this code was also generated using a crewai gpt for given use case. Use case is Narural Language to SQL

I could easily use other open source solution that can generate agent code for me. So I tried crewai gpt to do the same

You must understand that your user question “‘Name movie titles released in year 1945. Sort the listing by the descending order of movie’.”
Has NO semantic relationship too your Task description: “‘Interpret the user’s input and determine the relevant domain or workspace”, or the expected output "expected_output=‘The name of the domain (e.g., sales, finance, etc.) and any relevant metadata.’, You may as well ask your Task./Agent to tell what colour socks the user is wearing!

I am assuming that English is not your first language.

Let me give you a ‘hyperthetical’ example of what your Task/user question looks/sounds like.

If I said to you, “count the colours in this sentence?”, the sentence being: “What is the distance between the earth and he moon”! It would be ‘impossible’ for you to answer as there’s no ‘semantic’ similarity between what I ask you to do, and what I say!

Q: How do you expect any form of intellegence, artificial or human to aswere the question: "The name of the domain (e.g., sales, finance, etc.) and any relevant metadata… From the user input: “‘Name movie titles released in year 1945. Sort the listing by the descending order of movie” The only option you leave the LLM is to take a guess based on the options and expected output that you have given it.

The best answer I can give you is: You do not understand the basics of LLM’s, LLM prompts, prompt engineering to use CrewAI.

However: CrewAI is a great place to learn these things as it provides the easiest evironment to do so. You can’t ‘copy & paste’ your way through life. Honestly, spend a bit time learning about LLM prompting, the relationship between CrewAI prompts. It has the capacity to change your life :grinning:

This is FREE course presented by the founder of CrewAI. Take a couple of hours, it’s worth it :grinning:

There’s no open source solution that would give you exactly what you need. It may give you a close solution. You need to do some work yourself!

Its working now, there was no problem in the Agent or Task Defintion except I forgot to put parenthesis around user input .thanks for lecturing me but not helping me identify the problem…

Agent: Intent Recognition Agent

Task: Interpret the Get a list of movie titles released in year 1945. Sort the listing by the descending order of movie and determine the relevant domain or workspace.

Agent: Intent Recognition Agent

Final Answer:

The relevant domain for the request to “Get a list of movie titles released in year 1945. Sort the listing by the descending order of movie” is the “Entertainment” domain. The relevant metadata for this domain includes:

  • Movie Title
  • Release Year
  • Genre
  • Director
  • Main Cast
  • Rating
    This schema will allow you to retrieve and organize the list of movie titles from the year 1945 in descending order.

Agent: Table Selector Agent

Task: Based on the user’s intent, identify the relevant tables from the database schema. Provide the user an opportunity to review and adjust table selection.

Agent: Table Selector Agent

Final Answer:

Relevant Tables:

  1. Movies

Where Clause from the database schema:

SELECT Movie_Title 
FROM Movies 
WHERE Release_Year = 1945 
ORDER BY Movie_Title DESC;

You must be happy to know that crew ai gpt generated a working code , except it didnt format the variables with a {}…

Yes you figured it out. I faced the same think. For me Crew absolutely took over and started generating its own input ignoring what i have give in the kickoff function. Finally i figured out that what ever you have given in the kickoff function
{“inputparam”:“values”} must be embedded into your agent or task goal or description. What surprised me is this not clear in any of the documentations. I would request crewAI team to document these things upfront.

While I am pleased that you got it working, I still stand by my comments based on what you ‘disclosed’. I still recommend that you take some time and view the free course that I mention above.
If you had declared that your crew input was a DB query my responses would have been different.

I did miss the ‘missing opening curly brace’ :roll_eyes:, however the error was not related to your question of the Agent hallucinating.

‘Lectureing’, honest advice!

Best of luck with your project.

1 Like

Not only do the course but also read the Docs - Introduction - CrewAI