Why am I getting the "Invalid response from LLM call - None or empty" error with my custom tool if using Anthropic LLM but not with OpenAI LLM?

[dave-propeller]

Hi,

After upgrading to the latest version and migrating my anthropic setup from ChatAnthropic() to LLM() - I now recieve this error after a couple of successful tool calls.

fastapi_1       | # Agent: Sql Agent
fastapi_1       | ## Task: Taking question rank our sales agents performance from last month breaking down won and lost, and including revenue? - find the correct tables and create a query to find the answer
fastapi_1       | 
fastapi_1       | 
fastapi_1       | # Agent: Sql Agent
fastapi_1       | ## Thought: To answer this question, I need to find the correct tables and create a query that ranks our sales agents' performance from last month, including won and lost opportunities, and revenue. Let's start by identifying the relevant tables and their schemas.
fastapi_1       | ## Using tool: Tool for getting available tables names
fastapi_1       | ## Tool Input: 
fastapi_1       | "{}"
fastapi_1       | ## Tool Output: 
fastapi_1       | ['intranet.crm_opportunities', 'intranet.crm_opportunities_services', 'easy_pqq.b_clients_pqq', 'propeller_intranet_2007.a_clients', 'propeller_intranet_2007.a_contacts']
fastapi_1       | 
fastapi_1       | 
fastapi_1       | # Agent: Sql Agent
fastapi_1       | ## Thought: Thought: Based on the available tables, we'll need to use intranet.crm_opportunities for opportunity data, propeller_intranet_2007.a_contacts for sales agent information, and possibly intranet.crm_opportunities_services for revenue data. Let's check the schema of these tables.
fastapi_1       | ## Using tool: Get Schema Information
fastapi_1       | ## Tool Input: 
fastapi_1       | "{\"table_name\": \"intranet.crm_opportunities\"}"
fastapi_1       | ## Tool Output: 
fastapi_1       | CREATE TABLE `crm_opportunities` (
fastapi_1       |   `opportunity_id` int unsigned NOT NULL COMMENT 'Unique Identifier for each sale or opportunity',
fastapi_1       |   `client_id` bigint unsigned NOT NULL,
fastapi_1       |   `db_client_id` bigint unsigned NOT NULL,
fastapi_1       |   `pqq_client_id` int unsigned NOT NULL,
fastapi_1       |   `pqq_office_id` int unsigned DEFAULT NULL,
fastapi_1       |   `pqq_contact_id` int unsigned DEFAULT NULL,
fastapi_1       |   `created_by_id` int unsigned DEFAULT NULL,
fastapi_1       |   `created_by_ghost` tinyint unsigned DEFAULT '0',
fastapi_1       |   `status_id` int unsigned DEFAULT NULL,
fastapi_1       |   `opportunity_status_id` int unsigned DEFAULT NULL,
fastapi_1       |   `opportunity_status_group_id` int unsigned DEFAULT NULL,
fastapi_1       |   `opportunity_account_manager_id` bigint unsigned DEFAULT NULL,
fastapi_1       |   `enquiry_source_id` int unsigned DEFAULT NULL,
fastapi_1       |   `date_created` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
fastapi_1       |   `date_deadline` datetime DEFAULT '0000-00-00 00:00:00',
fastapi_1       |   `date_start` datetime DEFAULT '0000-00-00 00:00:00',
fastapi_1       |   `date_finish_estimated` datetime DEFAULT '0000-00-00 00:00:00',
fastapi_1       |   `opportunity_name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT 'Name of Sale or Opportunity',
fastapi_1       |   `opportunity_won_lost` int unsigned DEFAULT '0' COMMENT 'Success Status of Sale or Opportunity values are 0=pending,1=won,2=lost',
fastapi_1       |   `opportunity_won_lost_date` datetime DEFAULT '0000-00-00 00:00:00' COMMENT 'Date Success Status Changed',
fastapi_1       |   `client_ref` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   `opportunity_reference_number` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   `description` longtext COLLATE utf8mb4_general_ci,
fastapi_1       |   `hr_office_id` int unsigned DEFAULT '0',
fastapi_1       |   `hr_division_id` int unsigned DEFAULT '0',
fastapi_1       |   `structure_type_id` int unsigned NOT NULL DEFAULT '0',
fastapi_1       |   `tender_docs_due_out` datetime DEFAULT '0000-00-00 00:00:00',
fastapi_1       |   `tender_docs_received` tinyint unsigned NOT NULL DEFAULT '0',
fastapi_1       |   `date_tender_docs_received` datetime DEFAULT '0000-00-00 00:00:00',
fastapi_1       |   `is_tender_submitted` tinyint unsigned NOT NULL DEFAULT '0',
fastapi_1       |   `ol_address_1` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   `ol_address_2` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   `ol_address_3` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   `ol_address_4` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   `ol_town` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   `ol_county` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   `ol_postcode` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   `ol_country` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   `is_tender` int DEFAULT NULL,
fastapi_1       |   `known_competitor_id` int DEFAULT NULL,
fastapi_1       |   `lost_reason_id` int unsigned NOT NULL DEFAULT '0',
fastapi_1       |   `competitors_price` decimal(10,2) unsigned NOT NULL DEFAULT '0.00',
fastapi_1       |   `dev_notes` text COLLATE utf8mb4_general_ci,
fastapi_1       |   `linked_contract_id` int DEFAULT '0',
fastapi_1       |   `report_id` int NOT NULL DEFAULT '0' COMMENT 'contact report id',
fastapi_1       |   `archived` tinyint(1) DEFAULT '0',
fastapi_1       |   `date_archived` datetime DEFAULT NULL,
fastapi_1       |   `archived_by_id` int DEFAULT NULL,
fastapi_1       |   `opportunity_lost_notes` varchar(255) COLLATE utf8mb4_general_ci DEFAULT NULL,
fastapi_1       |   PRIMARY KEY (`opportunity_id`),
fastapi_1       |   KEY `client_id` (`client_id`),
fastapi_1       |   KEY `db_client_id` (`db_client_id`),
fastapi_1       |   KEY `pqq_client_id` (`pqq_client_id`),
fastapi_1       |   KEY `pqq_office_id` (`pqq_office_id`),
fastapi_1       |   KEY `pqq_contact_id` (`pqq_contact_id`),
fastapi_1       |   KEY `created_by_id` (`created_by_id`),
fastapi_1       |   KEY `created_by_ghost` (`created_by_ghost`),
fastapi_1       |   KEY `opportunity_status_id` (`opportunity_status_id`),
fastapi_1       |   KEY `opportunity_account_manager_id` (`opportunity_account_manager_id`),
fastapi_1       |   KEY `enquiry_source_id` (`enquiry_source_id`),
fastapi_1       |   KEY `opportunity_won_lost` (`opportunity_won_lost`),
fastapi_1       |   KEY `hr_office_id` (`hr_office_id`),
fastapi_1       |   KEY `hr_division_id` (`hr_division_id`),
fastapi_1       |   KEY `structure_type_id` (`structure_type_id`),
fastapi_1       |   KEY `tender_docs_received` (`tender_docs_received`),
fastapi_1       |   KEY `is_tender_submitted` (`is_tender_submitted`),
fastapi_1       |   KEY `is_tender` (`is_tender`),
fastapi_1       |   KEY `known_competitor_id` (`known_competitor_id`),
fastapi_1       |   KEY `lost_reason_id` (`lost_reason_id`),
fastapi_1       |   KEY `date_created` (`date_created`),
fastapi_1       |   KEY `date_deadline` (`date_deadline`),
fastapi_1       |   KEY `date_tender_docs_received` (`date_tender_docs_received`),
fastapi_1       |   KEY `date_start` (`date_start`),
fastapi_1       |   KEY `archived` (`archived`),
fastapi_1       |   KEY `status_id` (`status_id`),
fastapi_1       |   CONSTRAINT `crm_opportunities_ibfk_1` FOREIGN KEY (`pqq_client_id`) REFERENCES `easy_pqq`.`b_clients_pqq` (`pqq_client_id`),
fastapi_1       |   CONSTRAINT `crm_opportunities_ibfk_2` FOREIGN KEY (`opportunity_account_manager_id`) REFERENCES `propeller_intranet_2007`.`a_contacts` (`contact_id`),
fastapi_1       |   CONSTRAINT `crm_opportunities_ibfk_3` FOREIGN KEY (`db_client_id`) REFERENCES `propeller_intranet_2007`.`a_clients` (`client_id`),
fastapi_1       |   CONSTRAINT `crm_opportunities_ibfk_4` FOREIGN KEY (`status_id`) REFERENCES `crm_opportunities_statuses_rel` (`status_id`)
fastapi_1       | ) /*!50100 TABLESPACE `innodb_file_per_table` */ ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci ENCRYPTION='Y'
fastapi_1       | 
fastapi_1       | 
fastapi_1       | # Agent: Sql Agent
fastapi_1       | ## Thought: Thought: Now that we have the schema for crm_opportunities, we can see that it contains the necessary information for opportunities, including the account manager, won/lost status, and date created. Let's check the schema for intranet.crm_opportunities_services to get revenue information.
fastapi_1       | ## Using tool: Get Schema Information
fastapi_1       | ## Tool Input: 
fastapi_1       | "{\"table_name\": \"intranet.crm_opportunities_services\"}"
fastapi_1       | ## Tool Output: 
fastapi_1       | CREATE TABLE `crm_opportunities_services` (
fastapi_1       |   `data_id` int unsigned NOT NULL AUTO_INCREMENT,
fastapi_1       |   `opportunity_id` int unsigned NOT NULL,
fastapi_1       |   `service_id` int unsigned NOT NULL,
fastapi_1       |   `service_value` decimal(65,2) DEFAULT '0.00' COMMENT 'Service Value in Pounds (can be used to get total opportunity value)',
fastapi_1       |   `last_value_update_note` longtext COLLATE utf8mb4_general_ci,
fastapi_1       |   PRIMARY KEY (`data_id`),
fastapi_1       |   KEY `opportunity_id` (`opportunity_id`),
fastapi_1       |   KEY `service_id` (`service_id`),
fastapi_1       |   CONSTRAINT `crm_opportunities_services_ibfk_1` FOREIGN KEY (`opportunity_id`) REFERENCES `crm_opportunities` (`opportunity_id`),
fastapi_1       |   CONSTRAINT `crm_opportunities_services_ibfk_2` FOREIGN KEY (`service_id`) REFERENCES `easy_pqq`.`b_clients_pqq_crm_services` (`service_id`)
fastapi_1       | ) /*!50100 TABLESPACE `innodb_file_per_table` */ ENGINE=InnoDB AUTO_INCREMENT=4476452 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci ENCRYPTION='Y'
fastapi_1       | 
fastapi_1       | 
fastapi_1       | You ONLY have access to the following tools, and should NEVER make up tools that are not listed here:
fastapi_1       | 
fastapi_1       | Tool Name: Tool for getting todays date
fastapi_1       | Tool Arguments: {}
fastapi_1       | Tool Description: Get today's date, no input required, output is a string of todays date in the format of YYYY-MM-DD
fastapi_1       | Tool Name: Tool for getting available tables names
fastapi_1       | Tool Arguments: {}
fastapi_1       | Tool Description: Input is an empty string, output is a list of table names including schemas
fastapi_1       | Tool Name: Get Schema Information
fastapi_1       | Tool Arguments: {'table_name': {'description': None, 'type': 'str'}}
fastapi_1       | Tool Description: Get schema information from the specified table_name, input is the table_name
fastapi_1       | Tool Name: Get Example Rows from a table
fastapi_1       | Tool Arguments: {'table_name': {'description': None, 'type': 'str'}}
fastapi_1       | Tool Description: This tool provides example rows from table_name
fastapi_1       | Tool Name: Validate the SQL Generated
fastapi_1       | Tool Arguments: {'sql': {'description': None, 'type': 'str'}}
fastapi_1       | Tool Description: This tool will validate a SQL statement and return rows as json
fastapi_1       | 
fastapi_1       | Use the following format:
fastapi_1       | 
fastapi_1       | Thought: you should always think about what to do
fastapi_1       | Action: the action to take, only one name of [Tool for getting todays date, Tool for getting available tables names, Get Schema Information, Get Example Rows from a table, Validate the SQL Generated], just the name, exactly as it's written.
fastapi_1       | Action Input: the input to the action, just a simple python dictionary, enclosed in curly braces, using " to wrap keys and values.
fastapi_1       | Observation: the result of the action
fastapi_1       | 
fastapi_1       | Once all necessary information is gathered:
fastapi_1       | 
fastapi_1       | Thought: I now know the final answer
fastapi_1       | Final Answer: the final answer to the original input question
fastapi_1       | 
fastapi_1       |  Received None or empty response from LLM call.

It is odd that on the last tool call before the error message - that crew is injecting the Agent Tool Rules and Formatting again.

The same code and flow will work using openai models. So not sure why anthropic breaks in this way.

Any ideas on what I need to change?

Cheers
Dave

[vkrishna]

Seeing similar issues while trying to work with local hosted llms
tried with gemma-2b 4 Quant, ollama 3.2 8 Quant no luck.
And the tool calling is pretty bad on these as well.

[frieda-huang]

I have a similar issue regarding the error Received None or empty response from LLM call. I noticed that this occurs whenever self.messages in the following gets super long.

answer = self.llm.call(self.messages, callbacks=self.callbacks)

[rokbenko]

@vkrishna

tried with gemma-2b 4 Quant, ollama 3.2 8 Quant no luck.

@dave-propeller @vkrishna @frieda-huang Those are all small LLMs. Try to switch to a more capable LLM.

[vkrishna]

Thank you for the response. Do you have any recommendations of open source /free models please? Am doing this new so would appreciate any help with this.

[rokbenko]

@vkrishna

Do you have any recommendations of open source /free models please?

There are two general rules:

• The newer the LLM family, the better. For example, Llama 3.2 family is a better choice than Llama 3.1 family.
• The higher the LLM parameter number, the better. For example, Llama 3.2 11B is a better choice than Llama 3.2 3B.

There’s no formula to say a specific LLM will work. Simply try and see.

[rtk]

I have been experiencing the same issue as well using the newest version of Claude Sonnet 3.5 through Amazon Bedrock, so it didn’t seem like it was related to model capability.

The only thing I have found to work is setting Agent(..., use_system_prompt=False). That would be my first recommendation since you are also using Anthropic.

The system prompt does not seem to be playing nice with Anthropic but I am not sure why.

[rokbenko]

Looks like Anthropic LLMs, even the more capable ones like Claude 3.5 Sonnet, sometimes have difficulties and produce the Received None or empty response from LLM call error as @frieda-huang reported in another topic.

Custom tool throws error `ValueError: Invalid response from LLM call - None or empty`

I switched from claude-3-5-sonnet-20241022 to gpt-4o and it now works! Is there a way we can have a fallback option where if one model fails to accomplish the task, another equally or more capable model can pick it up? I belive LiteLLM has this feature (Reliability - Retries, Fallbacks | liteLLM)

In case you’re using one of the Anthropic LLMs, set the use_system_prompt parameter to False to the agent as @rtk suggested above, but that might cause some other issues down the road since system prompts in CrewAI exist to help agents to create the best final answer possible, not to cause issues. Or, you could simply switch the LLM provider to OpenAI, for example, if you’re LLM provider agnostic.

So, keep this in mind besides the two general rules I’ve suggested above:

@rokbenko

There are two general rules:

• The newer the LLM family, the better. For example, Llama 3.2 family is a better choice than Llama 3.1 family.
• The higher the LLM parameter number, the better. For example, Llama 3.2 11B is a better choice than Llama 3.2 3B.

There’s no formula to say a specific LLM will work. Simply try and see.

[rokbenko]

@dave-propeller

The same code and flow will work using openai models. So not sure why anthropic breaks in this way.

Will let the CrewAI staff know to see if there’s anything they could do to mitigate this issue with Anthropic LLMs.

[achris7]

@rokbenko how about gemini models? getting several errors trying different configurations. thanks

[achris7]

@rokbenko any insights re gemini models?

[rokbenko]

@achris7 No, unfortunatelly. I’m not part of CrewAI staff. I did as much as I could. That’s letting them know about the issue via a private channel and sharing the URL to this topic.

Which Gemini LLM exactly are you using?

[achris7]

Initially it was “pro” only via litellm but I’ve gotten 1.5 pro to work via open ai configuration . Thanks rok!

[rokbenko]

@achris7

Initially it was “pro” only via litellm but I’ve gotten 1.5 pro to work via open ai configuration . Thanks rok!

That’s exactly why I asked you what LLM you’re using. I wanted to suggest you switch to a more capable LLM if possible. Glad you solved the issue with the Gemini 1.5 Pro!