Update my LLM set-up after 0.60.0 update

General
1

For anyone who has looked, copied my code. Since updating to CrewAI 0.60.0 I use my llm’s like this:

import os
from langchain_community.llms import OpenAI, Ollama
from langchain_openai import ChatOpenAI
from langchain_groq import ChatGroq

# v < 0.60.0
@staticmethod
class LLMS:
    def __init__(self):
        self.OpenAIGPT35 = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0.7)
        self.OpenAIGPT4oMini = ChatOpenAI(model_name="gpt-4o-mini", temperature=0.8)
        self.OpenAIGPT4o = ChatOpenAI(model_name="gpt-4o", temperature=0.8)
        self.OpenAIGPT4 = ChatOpenAI(model_name="gpt-4", temperature=0.8)
        # self.Phi3 = Ollama(model="phi3:mini")
        self.Llama3_1 = Ollama(model="llama3.1")
        self.Phi3 = Ollama(model="phi3:medium-128k")
        # self.Phi3 = ChatOpenAI(model_name="phi3:medium-128k", temperature=0, api_key="ollama", base_url="http://localhost:11434")
        self.groqLama3_8B_3192 = ChatGroq(temperature=0.5, groq_api_key=os.environ.get("GROQ_API_KEY"),
                                          model_name="llama3-8b-8192")

# v >= 0.60.0
@staticmethod
class LLMS60:
    def __init__(self):
        self.OpenAIGPT35 = "gpt-3.5-turbo"
        self.OpenAIGPT4oMini = "gpt-4o-mini"
        self.OpenAIGPT4o = "gpt-4o"
        self.OpenAIGPT4 = "gpt-4"
        self.Phi3Mini = "ollama/phi3:mini"
        self.Llama3_1 = "ollama/llama3.1"
        self.Phi3Med ="ollama/phi3:medium-128k"
        self.groqLama3_8B_3192 = ChatGroq(temperature=0.5, groq_api_key=os.environ.get("GROQ_API_KEY"),
                                          model_name="llama3-8b-8192")

The changes are due to using LiteLLM which does a lot of the heavy lifting, makes it less complex for us :grin:

Here is my present ollama local models:

Screenshot from 2024-09-17 17-09-51
Screenshot from 2024-09-17 17-09-51743×381 80.8 KB

2

You do not even need to setup like that anymore just call them directly in the agent like so:

llm=‘groq/llama-3.1-8b-instant’

groq/llama-3.1-8b-instant

https://models.litellm.ai

2 Likes
3

I still think it nice to have a list of all your LLM models available after ‘self.’
A single data class that all of your Agents can access, a single point of maintenance, etc.

My old habits :exclamation:

You are correct that the simplest way is how you describe :slight_smile:

4

Can I ask how you assign temperature for an LLM like GPT-4o-mini when v ≥ 0.60.0?

And if v ≥ 0.60.0, can we still use an LLM like GPT-4o-mini with a configuration such as self.OpenAIGPT4oMini = ChatOpenAI(model_name=“gpt-4o-mini”, temperature=0.8)?

Thanks.

5

If you check out the input prms here for LiteLLM, it does suggest that it’s there!
@matt ?

1 Like
6

All you are doing is bloating your code by using langchain to still handle models.

You don’t need a data class any more as you can just set a string on the agent, you could just set the variable in the init instead I.E

llm=‘gpt4o’

This is why we removed langchain are now using LiteLLM instead, reduces lines required + less dependancies overall

I do hear you though, old habits etc

7

For parameters like temp and top_p can you try setting an env var like so

os.environ["temperature"] = "0.5"
os.environ["top_p"] = "0.1"

1 Like
8

but those need to be , by model, since we have several of them going at the same time. How can we do that? Also for lm studio models that use open api,

response = litellm.completion(
model=“openai/mistral”, # add openai/ prefix to model so litellm knows to route to OpenAI
api_key=“”, # api key to your openai compatible endpoint
api_base=“http://127.0.0.1:1234/v1” # set API Base of your Custom OpenAI Endpoint

)

Would that be correct?

9

By model meaning the same model multiple times or different models?

10

I’m looking into LM Studio

11

Thank you so much! :grin:

12

Thank you so much! :smile:

13

both. I use different models for general vs function calling vs planning. I may use the same model like gpt4omini but with different temperature parameters for different purposes. for example for a planning lm I always increase the temperature compared to using the same model for an agent that is just doing web search duty.

14

import os

from dotenv import load_dotenv

load_dotenv()

os.environ[“OPENAI_API_KEY”] = os.getenv(“OPENAI_API_KEY”)

os.environ[“temperature”] = “0.0”

Is this setup correct for the API key and temperature?

15

Wow - yeah my code totally broke too. I’m down for the Litellm use (it’s great and just works) but like others have mentioned I need to be able to pass in parameters to each model.

A more traditional litellm completion might look like:

response = completion(model=model_name,
messages=messages,
temperature=1.25,
top_p=0.7,
top_k=85,
max_tokens=5000,
frequency_penalty=0.4,
presence_penalty=0.2,
fallbacks=[“openrouter/anthropic/claude-3-haiku:beta”]
stop=[“[STOP]”]
)

how can I pass in parameters to the llm= value?

16

The only way I can get gemini pro and flash to work now is by installing google.generative ai.
Still haven’t figured out lm studio.

1 Like
17

I haven’t figured out a better way, but I got flash to work this way (Google API Key in .env file):

import litellm

load_dotenv()
litellm.api_key = os.getenv(‘GOOGLE_API_KEY’)

llm=‘gemini/gemini-1.5-flash-exp-0827’,

1 Like
18

Yes, I had that working but wanted to initialialize gemini with parameters…

19

Yeah. In the same boat.

20

May I ask if os.environ[“temperature”] = “0.0” and os.environ[“top_p”] = “0.1” are working with Gemini?