Agent Delegation Issue: CrewAI Skipping Missing Field Handling for URL Agent and Jumping to Summary Generation

Despite configuring my agents and tasks to allow communication and delegation for missing field resolution, the process skips directly to the summarization stage after identifying missing fields, instead of delegating back to the URL Validator and Researcher for further resource gathering. I am using gpt-4o-mini

Here’s the scenario:

  1. Agent 1: URL Validator and Researcher - Finds and validates URLs for plant species research.
  2. Agent 2: Plant Species Data Collector - Extracts data from the validated URLs and should delegate back to Agent 1 to fill missing fields.
  3. Agent 3: Plant Summary Generator - Summarizes the collected data into a paragraph.

When the Plant Species Data Collector identifies missing fields (e.g., “chemical constituents,” “pharmacological effects”), the logs show the LLM correctly concluding it should request additional resources from Agent 1. However, instead of delegating back, the workflow jumps directly to Agent 3 for summarization, producing incomplete results.

Code:

# Initialize LLMs
llm_1 = ChatOpenAI(model="gpt-4o-mini-2024-07-18", temperature=0, max_retries=3, max_tokens=None)
llm_2 = ChatGroq(model="groq/llama-3.1-8b-instant", temperature=0, max_tokens=None)

# Initialize tools
search_tool = SerperDevTool()
scrape_tool = ScrapeWebsiteTool()

# Agent 1: URL Validator and Researcher
url_validator_agent = Agent(
    role="URL Validator and Researcher",
    goal=(
        "Find and validate reliable URLs for plant species research. "
        "If requested by the Plant Species Data Collector"
        "search specifically for missing fields and validate URLs against strict scientific criteria."
    ),
    tools=[search_tool],
    verbose=True,
    backstory=(
        "An expert researcher specialized in finding and validating authoritative "
        "sources for botanical and pharmacological information. Your task is to identify "
        "and verify credible URLs that provide comprehensive information about a specific plant species. "
        "Collaborate with the 'Plant Species Data Collector' to address missing fields by finding new sources. "
    ),
    llm=llm_1,
)

# Task 1: Find and Validate URLs
url_validation_task = Task(
    description=(
        "For the plant species named {plant_name}, use advanced search techniques to find "
        "the most reliable and informative URLs. Your goals are to:"
        "1. Find at least 3-5 credible URLs using SerperDev search."
        "2. Validate each URL against strict criteria:"
        "   - Must be from scientific, botanical, or medical sources."
        "   - Contain relevant information about the species."
        "   - Free from obvious misinformation."
        "**NOTE:** If requested by the 'Fields Validator Agent' or 'Plant Species Data Collector', "
        "focus your search on URLs specifically addressing the missing fields they identified."
    ),
    agent=url_validator_agent,
    expected_output=(
        "A list of validated, high-quality URLs for the specified plant species. "
        "If the request is field-specific, include a summary of the fields addressed by each URL."
    )
)


# Agent 2: Plant Data Collector
plant_researcher = Agent(
    role="Plant Species Data Collector",
    goal=(
        "Extract comprehensive pharmacognosy data from validated URLs. "
        "If certain information is missing after scraping, request additional sources from the 'URL Validator and Researcher'."
    ),
    tools=[scrape_tool],
    verbose=True,
    backstory=(
        "A meticulous researcher specializing in pharmacognosy, skilled at extracting detailed botanical and medicinal information."
    ),
    llm=llm_1,
    allow_delegation=True
)

# Task 2: Collect Detailed Plant Data
plant_info_task = Task(
    description=(
        "For the plant species named {plant_name}, extract comprehensive data from validated URLs. "
        "Ensure the following fields are populated:"
        "- Scientific name"
        "- Common names"
        "- Family, genus, species"
        "- Native habitat"
        "- Medicinal uses"
        "- Chemical constituents"
        "- Pharmacological effects"
        "- Preparation methods"
        "- Side effects"
        "- Geographical availability"
        "- Description"
        "- References"
        "Validate all collected data using the PlantInfoPharmacognosy model."
        "**NOTE:** If any data is missing, explicitly state the missing fields and request URL Validator and Researcher to find new sources for missing fields."
    ),
    agent=plant_researcher,
    expected_output="Structured data validated against the PlantInfoPharmacognosy model.",
    output_json=PlantInfoPharmacognosy,
    context=[url_validation_task],
    output_file="plant_summary.json"
)

# Agent 3: Summarization Agent
plant_summary_agent = Agent(
    role="Plant Summary Generator",
    goal="Summarize plant data into a detailed and consistent paragraph.",
    verbose=True,
    backstory=(
        "An expert in summarizing structured botanical data for pharmacognosy research. "
        "Your task is to create concise yet comprehensive summaries of plant data, focusing on the "
        "scientific name, common names, family, genus, species, native habitat, medicinal uses, "
        "chemical constituents, pharmacological effects, preparation methods, side effects, "
        "geographical availability, and a general description. You do not include references in the summary."
    ),
    llm=llm_2,
)

# Task 3: Generate Summary
plant_summary_task = Task(
    description=(
        "Using the structured plant data provided, generate a detailed and well-structured single paragraph without any headings summarizing the plant's "
        "scientific name, common names, family, genus, species, native habitat, medicinal uses, key chemical constituents, "
        "pharmacological effects, preparation methods, side effects, geographical availability, and a general description. "
        "Do not include the references field in the summary. Focus on providing a comprehensive summary of the plant's characteristics and uses."
    ),
    agent=plant_summary_agent,
    expected_output="A pharmacognosy-focused single detailed paragraph summarizing the plant data in markdown format, excluding the references field.",
    context=[plant_info_task],
    output_file="plant_summary.md"
)

# Crew Configuration
plant_research_crew = Crew(
    agents=[url_validator_agent, plant_researcher, plant_summary_agent],
    tasks=[url_validation_task, plant_info_task, plant_summary_task],
    verbose=True
)

plant_inputs = {'plant_name': "Adenostyles leucophylla (Willd.) Rchb."}
crew_result = plant_research_crew.kickoff(inputs=plant_inputs)

Workflow Log:

## Final Answer: 
Here are the validated, high-quality URLs for the plant species Adenostyles leucophylla:

1. [Wikipedia - Adenostyles](https://en.wikipedia.org/wiki/Adenostyles)
2. [World Flora Online - Adenostyles leucophylla](https://about.worldfloraonline.org/taxon/wfo-0000127981-2023-06)


# Agent: Plant Species Data Collector
## Task: For the plant species named Adenostyles leucophylla (Willd.) Rchb., extract comprehensive data from validated URLs. Ensure the following fields are populated:- Scientific name- Common names- Family, genus, species- Native habitat- Medicinal uses- Chemical constituents- Pharmacological effects- Preparation methods- Side effects- Geographical availability- Description- ReferencesValidate all collected data using the PlantInfoPharmacognosy model.**NOTE:** If any data is missing, explicitly state the missing fields and request URL Validator and Researcher to find new sources for missing fields.


# Agent: Plant Species Data Collector
## Final Answer: 
{
  "Scientific name": "Adenostyles leucophylla (Willd.) Rchb.",
  "Common names": [
    "Adenostyle a feuilles blanches",
    "Cavolaccio lanoso",
    "Filziger Alpendost"
  ],
  "Family, genus, species": {
    "Family": "Asteraceae",
    "Genus": "Adenostyles",
    "Species": "leucophylla"
  },
  "Native habitat": "Central Europe, primarily in the Alps.",
  "Medicinal uses": "Traditionally used in herbal medicine, though specific uses are not well-documented.",
  "Chemical constituents": "Information not available.",
  "Pharmacological effects": "Information not available.",
  "Preparation methods": "Information not available.",
  "Side effects": "Information not available.",
  "Geographical availability": "Native to Austria, France, Italy, and Switzerland.",
  "Description": "Adenostyles leucophylla is a perennial flowering plant belonging to the Asteraceae family, characterized by its white, woolly leaves and flowering heads.",
  "References": [
    "Wikipedia - Adenostyles",
    "World Flora Online - Adenostyles leucophylla",
    "Kew Science - Adenostyles leucophylla",
    "SEINet Portal Network - Adenostyles leucophylla",
    "Global Biodiversity Information Facility (GBIF) - Adenostyles leucophylla"
  ]
} 

Missing fields: 
- Chemical constituents
- Pharmacological effects
- Preparation methods
- Side effects

I will request additional sources to fill in the missing fields.


# Agent: Plant Summary Generator
## Task: Using the structured plant data provided, generate a detailed and well-structured single paragraph without any headings summarizing the plant's scientific name, common names, family, genus, species, native habitat, medicinal uses, key chemical constituents, pharmacological effects, preparation methods, side effects, geographical availability, and a general description. Do not include the references field in the summary. Focus on providing a comprehensive summary of the plant's characteristics and uses.

# Agent: Plant Summary Generator
## Final Answer: 
Adenostyles leucophylla (Willd.) Rchb., also known as Adenostyle a feuilles blanches, Cavolaccio lanoso, or Filziger Alpendost, is a perennial flowering plant belonging to the Asteraceae family, characterized by its white, woolly leaves and flowering heads. Native to Central Europe, primarily in the Alps, this plant is traditionally used in herbal medicine, though specific uses are not well-documented. The chemical constituents of Adenostyles leucophylla are not available, and its pharmacological effects are also unknown. Preparation methods for this plant are not available, and potential side effects have not been documented. Adenostyles leucophylla is native to Austria, France, Italy, and Switzerland, and its description as a perennial flowering plant highlights its unique characteristics.