Hi!
I have Llama3.1 (ollama-based) running on a remote server, accessible via an HTTP API (a header “Authorization” must be passed to the requests). I’m facing a problem with using it as an LLM for my Agent.
This request works fine:
curl -X 'POST' \
''LLAMA_URL'' \
-H 'accept: application/json' \
-H 'Authorization: 'LLAMA_AUTHORIZATION_TOKEN'' \
-H 'Content-Type: application/json' \
-d '{
"model": "llama3.1",
"prompt": "Why is the sea blue?"
}'
But running the following code
@agent
def researcher(self) -> Agent:
return Agent(
config=self.agents_config['researcher'],
# tools=[MyCustomTool()], # Example of custom tool, loaded on the beginning of file
verbose=True,
llm=LLM(model="ollama/llama3.1",
base_url=os.environ.get('LLAMA_URL')
api_key=os.environ.get('LLAMA_AUTHORIZATION_TOKEN'))
)
I get litellm.exceptions.APIConnectionError: litellm.APIConnectionError: OllamaException - {"message": "Token is missing or invalid"}
The token is 100% correct
I have also tried
from langchain_community.llms.ollama import Ollama
@agent
def researcher(self) -> Agent:
return Agent(
config=self.agents_config['researcher'],
verbose=True,
llm=Ollama(model="llama3.1", #also tried "ollama/llama3.1"
base_url=os.environ.get('LLAMA_URL')
headers={"Authorization": f"{os.environ.get('LLAMA_AUTHORIZATION_TOKEN')}"})
)
But in that case I get litellm.exceptions.BadRequestError( # type: ignore provider you are trying to call. You passed model=Ollama Params: {'model': 'llama3.1', 'format': None, 'options': {'mirostat': None, 'mirostat_eta': None, 'mirostat_tau': None, 'num_ctx': None, 'num_gpu': None, 'num_thread': None, 'num_predict': None, 'repeat_last_n': None, 'repeat_penalty': None, 'temperature': None, 'stop': None, 'tfs_z': None, 'top_k': None, 'top_p': None}, 'system': None, 'template': None, 'keep_alive': None, 'raw': None} Pass model as E.g. For 'Huggingface' inference endpoints pass in
completion(model=‘huggingface/starcoder’,…) Learn more: https://docs.litellm.ai/docs/providers
Could you please give me a hint on what I should change so it works?