Using Custom LLM Request

General
,
1

Hello everyone, I would like some help to build agents that use a custom LLM service.

I want to use a microservice that has a custom API to make requests to proprietary LLMs (OpenAI, Claude, Gemini, etc.).

This microservice has its own authentication, which makes it necessary to add a custom header and thus build a request instead of using the crewai.LLM method.

I tried to use the CustomLLM from litellm, but I got the following result:

from litellm import CustomLLM

class MyCustomLLM(CustomLLM):
    def completion(self, *args, **kwargs) -> litellm.ModelResponse:
      server_url = "https://localhost:8080/orchestration-api"
      headers = {
          "Authorization": f"Bearer {os.environ.get('ORCHESTRATION_API')}"
      }

      payload = {
          "model": kwargs["model"],
          "messages": kwargs["messages"]
      }

      response = requests.post(server_url, headers=headers, json=payload)
      response_data = response.json()
      print(response_data)
      response_data = response_data["choices"][0]["message"]["content"].strip()
      
      # .. convert to litellm.ModelResponse() and return
      return model_response

my_custom_llm = MyCustomLLM()

litellm.custom_provider_map = [
      {"provider": "my-custom-llm", "custom_handler": my_custom_llm}
  ]

However, when making the request to the agent, I receive the following error:

Screenshot 2024-11-05 at 23.34.44
Screenshot 2024-11-05 at 23.34.441836×1396 364 KB

Has anyone implemented something similar?

2
  • Missing Token: The Authorization header is missing a valid token, as indicated by "Bearer " without a token value.
  • Solution: Ensure that the ORCHESTRATION_API environment variable is properly set with a valid token.
  • Connection Error: The server endpoint (https://localhost:8080/orchestration-api) could not be reached.
  • Solution: Verify that the server is running, reachable, and that the URL is correct.
3

@MAllef hi! Have you managed to fix this? Could you give me a hint please?

3 Likes
4

Hey! Checking if somebody found solution to this?

5

Hello, checking if someone has solved this issue.