Failed to convert text into a Pydantic model with private LLM

[adriancappe]

Hello all,

I have configured a crew to "list files from a Sharepoint site, then read that file and lastly provide an executive summary. all my tools are custom and my LLM is private (Azure).

the crew woks fine except when in in the task (and some of them) I set output_pydantic to a Class to get a structured output.

there is a very similar situation from another user filed as Bug in Github([BUG] Pydantic output with azure function calling models failed with litellm auth error · Issue #1572 · crewAIInc/crewAI · GitHub) , but no solution has been provided.

Here is the out just snipet of the error:
indent preformatted text by 4 spaces

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.


Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

---------------------------------------------------------------------------
AuthenticationError                       Traceback (most recent call last)
File /opt/anaconda3/lib/python3.12/site-packages/litellm/llms/openai/openai.py:707, in OpenAIChatCompletion.completion(self, model_response, timeout, optional_params, litellm_params, logging_obj, model, messages, print_verbose, api_key, api_base, api_version, dynamic_params, azure_ad_token, acompletion, logger_fn, headers, custom_prompt_dict, client, organization, custom_llm_provider, drop_params)
    706             else:
--> 707                 raise e
    708 except OpenAIError as e:

File /opt/anaconda3/lib/python3.12/site-packages/litellm/llms/openai/openai.py:634, in OpenAIChatCompletion.completion(self, model_response, timeout, optional_params, litellm_params, logging_obj, model, messages, print_verbose, api_key, api_base, api_version, dynamic_params, azure_ad_token, acompletion, logger_fn, headers, custom_prompt_dict, client, organization, custom_llm_provider, drop_params)
    622 logging_obj.pre_call(
    623     input=messages,
    624     api_key=openai_client.api_key,
   (...)
    630     },
    631 )
    633 headers, response = (
--> 634     self.make_sync_openai_chat_completion_request(
    635         openai_client=openai_client,
    636         data=data,
    637         timeout=timeout,
    638         logging_obj=logging_obj,
    639     )
    640 )
    642 logging_obj.model_call_details["response_headers"] = headers

.....
....
File /opt/anaconda3/lib/python3.12/site-packages/openai/_base_client.py:1064, in SyncAPIClient._request(self, cast_to, options, retries_taken, stream, stream_cls)
   1063     log.debug("Re-raising status error")
-> 1064     raise self._make_status_error_from_response(err.response) from None
   1066 return self._process_response(
   1067     cast_to=cast_to,
   1068     options=options,
   (...)
   1072     retries_taken=retries_taken,
   1073 )

AuthenticationError: Error code: 401 - {'error': {'message': 'Your authentication token is not from a valid issuer.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_issuer'}}

During handling of the above exception, another exception occurred:

OpenAIError                               Traceback (most recent call last)
File /opt/anaconda3/lib/python3.12/site-packages/litellm/main.py:1626, in completion(model, messages, timeout, temperature, top_p, n, stream, stream_options, stop, max_completion_tokens, max_tokens, modalities, prediction, audio, presence_penalty, frequency_penalty, logit_bias, user, response_format, seed, tools, tool_choice, logprobs, top_logprobs, parallel_tool_calls, deployment_id, extra_headers, functions, function_call, base_url, api_version, api_key, model_list, **kwargs)
   1620     logging.post_call(
   1621         input=messages,
   1622         api_key=api_key,
   1623         original_response=str(e),
   1624         additional_args={"headers": headers},
   1625     )
-> 1626     raise e
   1628 if optional_params.get("stream", False):
   1629     ## LOGGING
....
....
    43 if current_attempt < self.max_attempts:
     44     return self.to_pydantic(current_attempt + 1)
---> 45 raise ConverterError(
     46     f"Failed to convert text into a Pydantic model due to the following error: {e}"
     47 )

ConverterError: Failed to convert text into a Pydantic model due to the following error: litellm.BadRequestError: OpenAIException - Error code: 401 - {'error': {'message': 'Your authentication token is not from a valid issuer.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_issuer'}}

Here is the code (just part of it for simplicity):

class UseCaseSection(BaseModel):
    usecase_emoji: str = Field(description="Section emoji (e.g., 🔍, 📊, 🎯)")
    usecase_title: str = Field(description="Use Case title")
    usecase_content: str = Field(description="Describe what the use case can deliver, execute or acomplish")
    key_highlights: List[str] = Field(description="Describe benefits of Use case")


UseCaseAgent= Agent(
    role="IT Use Case Analyst",
    goal=("Identify and extract IT-related use cases from documeread_file_task"
          "Output should be in readable markdown documents"
         ),
    backstory=( "An expert in IT operations, familiar with networking, security,\ "
        "and compliance processes. Skilled in analyzing technical documents\ "
        "to extract key use cases such as device onboarding, configuration\ "
        "compliance, and software upgrades.\ "
        "Anything that an action or result is not Device or Network Service related, should not be included as use case.\ "
        
    ),
    allow_delegation=False,
    memory=memory_set,  # Enable memory to store file list for Agent 2   
	verbose=True,
    llm=bridgeit_llm
    
        
)


use_case_task = Task(
    name="Use Case Identification Task",
    description=(""" Read the provided document and identify IT-related use cases such as 
                'Device Onboarding,' 'Configuration Compliance,' 'Software Upgrade,' 
                or similar. Extract and provide a brief description of each use case """
                ),
    agent=UseCaseAgent,
    expected_output="A structured list of identified IT use cases with summaries.",
    output_pydantic=UseCaseSection,
    output_file="UseCaseReport.md",
    context= [read_file_task],
    sync_mode=True
)


crew = Crew(
    agents=[PopulateCredentials, FileBrowser, FileReader, UseCaseAgent, SummarizationAgent, Formatter],
    tasks=[assign_credentials_task, list_files_task, read_file_task,use_case_task, summarize_task, formatting_task],
    manager_agent=RequestHandler,
    process=Process.hierarchical,
    verbose=True,

    
)


print("Inputs to Crew:", params_agent)

result = crew.kickoff(inputs=params_agent)

Any guidance or ideas on where the problem might reside will greatly help.

I really appreciate any help you can provide.
Adrian

[adriancappe]

I did some additional digging and I happened to find the reason for the authenticating error. LiteLLM uses in some cases https://api.openai.com/v1/" instead of the api_base I have provided. For some reason, it does it when out_pydantic is set in the task associated with the agent.

How can I force to get the crew to use my private LLM api_base and not change it ??

Here is a good case…

POST Request Sent from LiteLLM:
curl -X POST \
https://myprivate.llm.com \
-d '{'model': 'gpt-4o', 'messages': [{'role': 'system', 'content': 'You are Handle request. You are a skilled Engineer to generate high-level executive summaries Note the documents might already have an Executive summary. Do not use that executive summary, build your own.\nEnsure the summary is properly formatted\nYour personal goal is: Process the user request: {user_request}\nYou ONLY have access to the following tools, 

HEre is the case where for some reason it decides to use https://api.openai.com/v1/:

POST Request Sent from LiteLLM:
curl -X POST \
https://api.openai.com/v1/ \
-d '{'model': 'gpt-4o', 'messages': [{'role': 'user', 'content': '{\n  "usecase_emoji": "🔧",\n  "usecase_title": "Device Onboarding",\n  "usecase_content": "The process of integrating a new device into the enterprise network. This includes establishing secure network access, applying initial configuration settings, and verifying the operational status of the device.",\n  "key_highlights": ["Establishing secure network access", "Applying initial configuration settings", "Verifying operational status"`

thanks in advance!
Adrian

[Fuad_Mefleh]

Could this issue help fix the problem?

Does `Crew` Pydantic output require OpenAI

The docs for Task say Outputs a Pydantic model object, requiring an OpenAI client. Only one output format can be set. Does that mean that I cannot use an opensource LLM and get a Pydantic output from a crew?

[pedrosimao]

@adriancappe

out_pydantic

This seems like a simple bug to fix. I wonder if a fix is already been pushed to the github repo…

[adriancappe]

After some additional debugging I was able to identify the problem: You have to define environment variables in the OS shell:

api_key=os.getenv("OPENAI_API_KEY")
api_base = os.getenv("OPENAI_API_BASE")
llm_model=os.getenv("OPENAI_MODEL_NAME")

Initially, I had only the api_key. in this case some LLM calls would work and some would not

I’ll expand in a later post all the details. But it seems a bug.