# How to use "function\_calling\_llm" for agent **URL:** \ **Category:** CrewAI Community Support\ **Created:** [November 29, 2024, 5:19pm UTC](https://community.crewai.com/t/how-to-use-function-calling-llm-for-agent/1721 "2024-11-29T17:19:37Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![miqin](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@miqin](https://community.crewai.com/u/miqin)\ **Post date:** [November 29, 2024, 5:19pm UTC](https://community.crewai.com/t/how-to-use-function-calling-llm-for-agent/1721/1 "2024-11-29T17:19:37Z")
Hi, I’m trying to set “function\_calling\_llm” to increase the hit rate on generating correct function arguments. I use a custom model as below but it doesn’t work. > function\_llm = LLM( > model=“openai/llama3”, > base\_url=“[http://localhost:8000/v1](http://localhost:8000/v1)”, > api\_key=‘sk\_1234’, > temperature=0.5, > top\_p=0.5, > max\_tokens=1024 > ) > @agent > def my\_agent(self) → Agent: > > ``` > return Agent( > config=self.agents_config['my_agent'], > verbose=True, > allow_code_execution=True, > function_calling_llm = function_llm > ) > > ``` Through debugging, I found this “function\_calling\_llm” module seems never to work. Here are some possible issues I found. 1. Custom LLM can not work because the api\_key and other parameters is not passed down to litellm for request. According to the “to\_pydantic” function in “InternalInstructor” class, only the “model” attribute is passed down, while the others such as ‘api\_key’ are discarded. > ``` > def to_pydantic(self): > messages = [{"role": "user", "content": self.content}] > if self.instructions: > messages.append({"role": "system", "content": self.instructions}) > model = self._client.chat.completions.create( > model=self.llm.model, response_model=self.model, messages=messages > ) > return model > > ``` 1. Errors happen in following “\_function\_calling” function in “ToolUsage” class. > ``` > tool_object = converter.to_pydantic() > calling = ToolCalling( > tool_name=tool_object["tool_name"], > arguments=tool_object["arguments"], > log=tool_string, # type: ignore > ) > > ``` The converter.to\_pydantic() returns a pydantic model which is not subscriptable. It is very strange that the following logic gets the attributes of “tool\_object” as dictionary. The crewai version is 0.80.0 and I’m not sure if this “function\_calling\_llm” is ready in this version. Am I using this correctly? Any suggestion for this? ---
**Author:** ![matt](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/matt/32/3_2.png) [@matt](https://community.crewai.com/u/matt)\ **Post date:** [November 30, 2024, 12:38am UTC](https://community.crewai.com/t/how-to-use-function-calling-llm-for-agent/1721/2 "2024-11-30T00:38:42Z")
> [@miqin](#): > > Hi, I’m trying to set “function\_calling\_llm” to increase the hit rate on generating correct function arguments. I use a custom model as below but it doesn’t work. > > > function\_llm = LLM( > > model=“openai/llama3”, > > base\_url=“[http://localhost:8000/v1”](http://localhost:8000/v1%E2%80%9D), > > api\_key=‘sk\_1234’, > > temperature=0.5, > > top\_p=0.5, > > max\_tokens=1024 > > ) > > @agent > > def my\_agent(self) → Agent: > > > > ```auto > > return Agent( > > > > ``` > > ``` > config=self.agents_config['my_agent'], > verbose=True, > allow_code_execution=True, > function_calling_llm = function_llm > ) > > ``` > > > ```auto > > > > ``` > > Through debugging, I found this “function\_calling\_llm” module seems never to work. Here are some possible issues I found. > > 1. Custom LLM can not work because the api\_key and other parameters is not passed down to litellm for request. According to the “to\_pydantic” function in “InternalInstructor” class, only the “model” attribute is passed down, while the others such as ‘api\_key’ are discarded. > > > ```auto > > def to_pydantic(self): > > > > ``` > > ``` > messages = [{"role": "user", "content": self.content}] > if self.instructions: > messages.append({"role": "system", "content": self.instructions}) > model = self._client.chat.completions.create( > model=self.llm.model, response_model=self.model, messages=messages > ) > return model > > ``` > > > ```auto > > > > ``` > > 1. Errors happen in following “\_function\_calling” function in “ToolUsage” class. > > > ```auto > > tool_object = converter.to_pydantic() > > > > ``` > > ``` > calling = ToolCalling( > tool_name=tool_object["tool_name"], > arguments=tool_object["arguments"], > log=tool_string, # type: ignore > ) > > ``` > > > ```auto > > > > ``` > > The converter.to\_pydantic() returns a pydantic model which is not > subscriptable. It is very strange that the following logic gets the attributes of > “tool\_object” as dictionary. > > The crewai version is 0.80.0 and I’m not sure if this “function\_calling\_llm” is ready in this version. > > Am I using this correctly? Any suggestion for this? Thanks for raising this we are looking into this internally - will update this post once we have a solution for you ---
**Author:** ![matt](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/matt/32/3_2.png) [@matt](https://community.crewai.com/u/matt)\ **Post date:** [November 30, 2024, 12:40am UTC](https://community.crewai.com/t/how-to-use-function-calling-llm-for-agent/1721/3 "2024-11-30T00:40:41Z")
could you also please share any error messages you got? ---
**Author:** ![miqin](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@miqin](https://community.crewai.com/u/miqin)\ **Post date:** [November 30, 2024, 4:34am UTC](https://community.crewai.com/t/how-to-use-function-calling-llm-for-agent/1721/4 "2024-11-30T04:34:30Z")
Hi, @matt . Thanks for your reply. Here are the error messages I got. 1. In “to\_pydantic” function, “InternalInstructor” class > Traceback (most recent call last): > File “/Users/miqin/Library/Python/3.10/lib/python/site-packages/instructor/retry.py”, line 142, in retry\_sync > response = func(\*args, \*\*kwargs) > File “/Users/miqin/Library/Python/3.10/lib/python/site-packages/litellm/utils.py”, line 960, in wrapper > raise e > File “/Users/miqin/Library/Python/3.10/lib/python/site-packages/litellm/utils.py”, line 849, in wrapper > result = original\_function(\*args, \*\*kwargs) > File “/Users/miqin/Library/Python/3.10/lib/python/site-packages/litellm/main.py”, line 3059, in completion > raise exception\_type( > File “/Users/miqin/Library/Python/3.10/lib/python/site-packages/litellm/litellm\_core\_utils/exception\_mapping\_utils.py”, line 2136, in exception\_type > raise e > File “/Users/miqin/Library/Python/3.10/lib/python/site-packages/litellm/litellm\_core\_utils/exception\_mapping\_utils.py”, line 310, in exception\_type > raise AuthenticationError( > litellm.exceptions.AuthenticationError: litellm.AuthenticationError: AuthenticationError: OpenAIException - The api\_key client option must be set either by passing api\_key to the client or by setting the OPENAI\_API\_KEY environment variable 1. I guess the above error is caused by the base\_url and api\_key of my custom llm is not passed down. Therefore, I changed the source code and provide the base\_url and api\_key like that, > ``` > model = self._client.chat.completions.create( > model=self.llm.model,api_key =self.llm.api_key,api_base=self.llm.base_url,response_model=self.model, messages=messages > ) > > ``` Luckily, my custom llm is successfully called and response is returned normally. Then the other error happened in “\_function\_calling” function in “ToolUsage” class. > tool\_object = converter.to\_pydantic() > calling = ToolCalling( > tool\_name=tool\_object[“tool\_name”], > arguments=tool\_object[“arguments”], > log=tool\_string, # type: ignore > ) > Traceback (most recent call last): > File “/Users/miqin/Library/Python/3.10/lib/python/site-packages/IPython/core/interactiveshell.py”, line 3577, in run\_code > exec(code\_obj, self.user\_global\_ns, self.user\_ns) > File “”, line 2, in > tool\_name=tool\_object[“tool\_name”], > TypeError: ‘InstructorToolCalling’ object is not subscriptable Could you help check these issues? Very appreciate!