Does hierarchical process even work? Your experience is highly appreciated!

General
1

The sources of my doubts

All the hierarchical process topics I found have no responses:

I myself bumped into a blocking bug that reproduces itself reliably: Agents fail to call each other (and where to edit the system prompt)

Additionally, I found that the manager agent does not interpolate the query, so I see no way to make it decide which task is relevant to the current problem and which is not:

crew.py

  return Crew(
      tasks=self.tasks,
      agents=[ # cannot use self.agents because it includes self.coordinator()
          self.content_planner(),
          ...
          self.chief_editor(),
      ],
      manager_agent=self.coordinator(),
      process=Process.hierarchical,
      verbose=True,
  )

main.py

inputs['prompt'] = 'I am in a writing mood. Inspire me with some ideas for ongoing content pieces'
Publishing().crew().kickoff(inputs=inputs)

agentops log:

You are Content Planning Coordinator. You are a skilled coordinator who excels at prioritizing tasks and ensuring that the team's objectives are met.

Your personal goal is: {prompt}
You ONLY have access to the following tools...

...

Review the content backlog: <the first task from tasks.yaml. The task is of another agent>

It offers the coordinator agent a task that is explicitly connected to the content_planner agent. This gives us a clue :bulb: of how to make it start with the correct task. I’ve conducted that experiment too, but the results were even stranger: an implicit agent ‘Crew Manager’, defined by the framework, got the same task and the full list of tools of the task’s correct agent. So Crew Manager attempted to solve a task of Content Planner by himself. :see_no_evil:

Question :raising_hand_man:

Does someone run a crew that decides which tasks to use and which to bypass as non-relevant on its own?

Maybe you use something other than the hierarchical process?

P.S. I’ve completed ‘Multi AI Agent Systems with crewAI’ and ‘Practical Multi AI Agents and Advanced Use Cases with crewAI.’

There are lessons on hierarchical process, but the lessons do not render how those crews work. Following the course steps you simply:

  1. define a sequential process (this is the part where you still understand what you’re doing, thanks to preceding lessons)
  2. switch the param to hierarchical (stepping into terra incognita)
  3. pray that things will arrange themselves into something relevant (sometimes they do)
2

So far the only way I have been able to get hierarchical mode to work is by NOT defining a manager agent but just defining a manager_llm and that is all. It seems to take care of the rest.

	return Crew(
		agents=self.agents, # Automatically created by the @agent decorator
		tasks=self.tasks, # Automatically created by the @task decorator
		process=Process.hierarchical,
		verbose=True,
		manager_llm='ollama/llama3.2'
3

Thanks!

But does it always execute all the tasks, or is it picky (based on the inputs)?

If it executes all, then why not sequential - what’s the benefit?

Also, do you see error messages in the log like ‘cannot call another object, because the object {“description”: “…”, “type”: “str”} is not a string?’

4

I have limited experience and some of the bugs mentioned could be my own fault - but I will share the following observations:

-turning on verbose mode and piping all output to a text file can reveal much more good stuff that does not necessarily make it into the ‘final’ report stored by the manager. Reviewing that file will show what the Manager is trying to do and if it was successful at getting a response or not. Be sure to look at times when the manager has a ‘Thought’ about what it is doing. That is where I found some rich stuff.

-sometimes the interaction between the manager and a coworker does not go as planned; and instead, it will cycle or timeout leaving a task incomplete or returning with a very short answer. There is one reason, so far, that I observe this happening: that within the Manager request to delegate to a coworker with a task and a question at the same time, there is an error like the following:

Tool Output:

Error: the Action Input is not a valid key, value dictionary.

Unfortunately, this occurs when it is really getting going and getting deep into the problem. The task it was assigning and the question it was asking are really getting to the core of the problem and show it has learned from previous levels since the beginning.

But even after these errors, the interaction between the coworkers and the manager are impressive even considering I am still using a local LLM for testing and not deployed out to a production environment - once i was able to see all the stuff going on in verbose mode. Verbose mode was set here:

return Crew(
agents=self.agents,
tasks=self.tasks,
process=Process.hierarchical,
verbose=True,
manager_llm=‘ollama/llama3.2’

To answer the other part of your question, it seems to iterate in this mode more than in sequential mode, and I am seeing benefit as the manager parses answers from one coworker into questions for another coworker.

One additional note. Once I switched the crew LLM over from a local testing ollama to a production Azure hosted OpenAI model, the behavior of the Manager agent changed remarkably. It was clear that the Manager was really struggling with the format of the schemas and even developed a Thought that it was having a recurring problem.

“Agent: Crew Manager
Thought: Since previous attempts to delegate the work have failed, I will now simplify the descriptions even further, ensuring the language is direct and clear.”

“Agent: Crew Manager
Thought: Since past attempts to delegate the task have failed due to ongoing validation errors, I need to ensure the input strings are as clear and direct as possible. I will focus on making everything concise and precise.”

The larger LLM still gave a better result, and was orders of magnitude faster, but it was interesting that the delegating-of-tasks was hanging up so much. In this case the Manager asked much less questions of the coworkers and just kept trying to adjust the delegation to make the handoff work.

Hope this is helpful.

5

One more note - the errors the manager is getting while trying to delegate a task to a coworker appears related to it not following the format of that tool:
‘coworker’: {‘description’: ‘The role/name of the coworker to ask’, ‘type’: ‘str’}

Instead it is passing ‘coworker’ : ‘name_of_coworker’ as the input to the tool.