Using Phi3:medium (128k) as a manger LLM in Hierachical process issues

CrewAI Community Support
1

Given that Phi3 appears to have one of the best ‘reasoning’ scores amongst open-source LLM’s I am looking at using Phi3 as the LLM for the manager within a hierarchical process.
I have ollama running on my Ubuntu/Linux box./ I have pulled ‘phi3:14b’
I use this static class to test various LLMs with various agents, etc.:

@staticmethod
class LLMS:
    def __init__(self):
        self.OpenAIGPT35 = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0.7)
        self.OpenAIGPT4oMini = ChatOpenAI(model_name="gpt-4o-mini", temperature=0.8)
        self.OpenAIGPT4 = ChatOpenAI(model_name="gpt-4", temperature=0.8)
        self.Llama3_1 = Ollama(model="llama3.1")
        # self.Phi3 = Ollama(model="phi3:mini")
        self.Phi3 = ChatOpenAI(model_name="phi3:medium-128k", temperature=0, api_key="ollama", base_url="http://localhost:11434")
        self.groqLama3_8B_3192 = ChatGroq(temperature=0.5, groq_api_key=os.environ.get("GROQ_API_KEY"), model_name="llama3-8b-8192")

When I use Phi3 as the LLM for the manager nothing seems to work, I get errors like: " [2024-09-11 10:19:51][DEBUG]: == [Crew Manager] Task output: Agent stopped due to iteration limit or time limit." here’s my crew set-up:

online_business_model_crew = Crew(
    agents=[online_market_analyst_agent, website_structure_agent],
    tasks=[online_market_analysis_task, website_structure_task],
    manager_llm=llms.Phi3,
    process=Process.hierarchical,
    verbose=True
)

Can anybody suggest what cold be wrong with what I’m doing here?

2

Try setting the max iterations on the agent to a lower number

CleanShot 2024-09-11 at 18.43.14@2x
CleanShot 2024-09-11 at 18.43.14@2x2076×158 16.9 KB

3

Why would that help? Just seems like it would end up with the same problem, just quicker if you reduce iterations.

4

UPDATE:
If I set all Agent & Crew LLMs → GTP4omini all is OK

if I set Phi3.5 as manager LLM in crew then it starts, but then I get errors unable to find other agents in the crew, or externam tools (SuperDev) stops working with what look like mising arg errors.

So it is something to do with Phi3. I’ve tried Phi3-mini, medium-128K & instruct. All with basically the same types of issues as described above.

Phi3.5 is used via ollama

5

Because it is hitting the agent iteration limit which is 25 by default so setting a low number might force the agent to finish correctly

6

If its only happening to the one LLM then you might want to look at running agentops or something similar to see whats happening

7

Interesting idea. I have always found its the llm when iterations max although for simple tasks I always set the limit lower.
Nexusraven v2 always seems to work for function calling with strict sys instruction for json output on tool requests.
@Dabnis is it getting stuck on tool calling? On the superdev issue, do you give it sys prompt to make requests in valid json? Do you use separate llm’s for general agent inference vs tool calling? Is it timing out when you use sequential?

8

Hi Moto,
re " do you give it sys prompt to make requests in valid json?" This is new to me, could you please explain?
As for tool calling: Only Phi3 models show this issue, when I switch back to GTP4omini, tooling works OK.
FYI: I’ve tried Phi3:mini, Phi3:medium-128k & Phi3-instruct.
I will try it in sequencial mdoe & update here.

@matt ‘agentops’ is yet another new area a CrewAI for me, I’ll investigate.

10

Pretty sure those are just llms being dumb problems.
On the system prompt, I just add, you MUST use valid JSON format when you use tools or call functions. Some llm endpoints allow you to specify json output schema e.g. gemini flash, etc. In the previous version of LM Studio you could force an llm to output json.

11

Thanks for the info moto :pray: