# Crewai + multimodal

**URL:** <https://community.crewai.com/t/crewai-multimodal/1643>\
**Category:** CrewAI Community Support\
**Tags:** crewai, feature\
**Created:** [November 26, 2024, 5:53pm UTC](https://community.crewai.com/t/crewai-multimodal/1643 "2024-11-26T17:53:15Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paarttipaabhalaji](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/paarttipaabhalaji/32/2053_2.png) [@Paarttipaabhalaji](https://community.crewai.com/u/Paarttipaabhalaji)\
**Post date:** [November 26, 2024, 5:53pm UTC](https://community.crewai.com/t/crewai-multimodal/1643/1 "2024-11-26T17:53:16Z")

</div>

Hi Team,

I need to use the multimodal LLM in crewai. In case of multimodal, usually image need to convert into base64 image and send that encoded value into LLM param as “Image\_url” in message request body. I would like to know whether crewai have an options to send that image encoded value vis crewai’s LLM() package.

Here is my request body:

```
request_body_sample = {
    "messages": [{"role":"system","content":system_prompt}, {"role":"user","content":[{"type":"text","text":user_text_input},{"type":"image_url","image_url":{"url": f"data:image/jpeg;base64,{imgBase64EncValue}"}}]}],
    "project_id": credentials.get("project_id"),
    "model_id": "watsonx/meta-llama/llama-3-2-90b-vision-instruct",
    "decoding_method": "sample",
    "random_seed": 568743,
    "temperature": 0,
    "top_k": 50,
    "top_p": 1,
    "repetition_penalty": 1,
    "max_tokens": 8000
}    

response = requests.post(
    credentials.get("url"),
    headers=headers,
    json=request_body_sample
    )
if response.status_code != 200:
    raise Exception("Non-200 response: " + str(response.text))
data = response.json()
print(data['choices'][0]['message']['content'])

```

Or please help me to use multimodal in crewai in better way.

when I ask the question related to “multimodal support” documentation chatbot. Its output the message like

 ![image](https://us1.discourse-cdn.com/flex025/uploads/crewai/original/2X/8/88d7254d95d29e0b2f9745040a0064c747117ac8.jpeg)

crewai doesn’t support multimodal?

Thanking you.

---

<div class="post-metadata">

**Author:** ![uma-08](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/uma-08/32/1153_2.png) [@uma-08](https://community.crewai.com/u/uma-08)\
**Post date:** [November 30, 2024, 3:10pm UTC](https://community.crewai.com/t/crewai-multimodal/1643/2 "2024-11-30T15:10:22Z")

</div>

hey @Paarttipaabhalaji did you find a solution for this?

---

<div class="post-metadata">

**Author:** ![Paarttipaabhalaji](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/paarttipaabhalaji/32/2053_2.png) [@Paarttipaabhalaji](https://community.crewai.com/u/Paarttipaabhalaji)\
**Post date:** [December 2, 2024, 4:34am UTC](https://community.crewai.com/t/crewai-multimodal/1643/3 "2024-12-02T04:34:36Z")

</div>

No @uma-08 , please help me on this.

---

<div class="post-metadata">

**Author:** ![uma-08](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/uma-08/32/1153_2.png) [@uma-08](https://community.crewai.com/u/uma-08)\
**Post date:** [December 2, 2024, 6:52pm UTC](https://community.crewai.com/t/crewai-multimodal/1643/4 "2024-12-02T18:52:13Z")

</div>

sure would love to discuss workflows to execute on this, I’m also stuck in this. what’s the best way to connect with you, @Paarttipaabhalaji ?

---

<div class="post-metadata">

**Author:** ![Paarttipaabhalaji](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/paarttipaabhalaji/32/2053_2.png) [@Paarttipaabhalaji](https://community.crewai.com/u/Paarttipaabhalaji)\
**Post date:** [December 3, 2024, 6:37am UTC](https://community.crewai.com/t/crewai-multimodal/1643/5 "2024-12-03T06:37:23Z")

</div>

kindly connect me in linkedin.

---

<div class="post-metadata">

**Author:** ![matt](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/matt/32/3_2.png) [@matt](https://community.crewai.com/u/matt)\
**Post date:** [December 3, 2024, 12:02pm UTC](https://community.crewai.com/t/crewai-multimodal/1643/6 "2024-12-03T12:02:55Z")

</div>

We do sort of support multi-model through one of our tools - [Vision Tool - CrewAI](https://docs.crewai.com/tools/visiontool) but this is only a tool so it may or may not help

---

<div class="post-metadata">

**Author:** ![Paarttipaabhalaji](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/paarttipaabhalaji/32/2053_2.png) [@Paarttipaabhalaji](https://community.crewai.com/u/Paarttipaabhalaji)\
**Post date:** [December 4, 2024, 6:01am UTC](https://community.crewai.com/t/crewai-multimodal/1643/7 "2024-12-04T06:01:53Z")

</div>

@matt I need to use multimodal from the provider watsonx.

---

<div class="post-metadata">

**Author:** ![Colin\_Ng](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/colin_ng/32/1330_2.png) [@Colin\_Ng](https://community.crewai.com/u/Colin_Ng)\
**Post date:** [December 5, 2024, 5:21am UTC](https://community.crewai.com/t/crewai-multimodal/1643/8 "2024-12-05T05:21:25Z")

</div>

I also faced the same issue.

I want to directly input the image path to my local multimodal model, without using additional tool

---

<div class="post-metadata">

**Author:** ![Paarttipaabhalaji](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/paarttipaabhalaji/32/2053_2.png) [@Paarttipaabhalaji](https://community.crewai.com/u/Paarttipaabhalaji)\
**Post date:** [December 18, 2024, 12:02pm UTC](https://community.crewai.com/t/crewai-multimodal/1643/9 "2024-12-18T12:02:45Z")

</div>

@matt any guidance or update on this.

---

<div class="post-metadata">

**Author:** ![MayInRain](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/mayinrain/32/1840_2.png) [@MayInRain](https://community.crewai.com/u/MayInRain)\
**Post date:** [January 6, 2025, 12:48pm UTC](https://community.crewai.com/t/crewai-multimodal/1643/10 "2025-01-06T12:48:32Z")

</div>

Same here! I’m trying to work with two images using a multimodal model, but can only send them as base64 strings.

---

<div class="post-metadata">

**Author:** ![BradLeon](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/bradleon/32/3346_2.png) [@BradLeon](https://community.crewai.com/u/BradLeon)\
**Post date:** [April 9, 2025, 2:48pm UTC](https://community.crewai.com/t/crewai-multimodal/1643/11 "2025-04-09T14:48:08Z")

</div>

same issue, looking forward a solution.

---

<div class="post-metadata">

**Author:** ![maxmoura](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/maxmoura/32/4206_2.png) [@maxmoura](https://community.crewai.com/u/maxmoura)\
**Post date:** [April 10, 2025, 10:44am UTC](https://community.crewai.com/t/crewai-multimodal/1643/12 "2025-04-10T10:44:50Z")

</div>

It’d be great if you could provide a detailed description of what you’re trying to accomplish, how you’re attempting to do it, and ideally, include a short code snippet. That way, you’ll have a much better shot at getting the help you need.

---

<div class="post-metadata">

**Author:** ![netcmcc](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/netcmcc/32/3365_2.png) [@netcmcc](https://community.crewai.com/u/netcmcc)\
**Post date:** [April 10, 2025, 11:50am UTC](https://community.crewai.com/t/crewai-multimodal/1643/13 "2025-04-10T11:50:54Z")

</div>

> <https://github.com/crewAIInc/crewAI/issues/2565>
>
> \### Description
> 
> The multimodality feature in CrewAI v0.114.0 does not properly …handle image inputs when sending requests to LLMs. The agent fails to correctly format and send image data to the LLM, resulting in incomplete or failed image analysis tasks. 
> 
> Because the image was not submitted to LLM in the format agreed by OpenAI API, the image type task did not work properly. See the log for details.
> 
> https://platform.openai.com/docs/guides/images?format=base64-encoded#provide-multiple-image-inputs
> https://docs.anthropic.com/en/docs/build-with-claude/vision
> 
> 
> 
> \### Steps to Reproduce
> 
> 1. Create an agent with \`multimodal=True\`
> 2. Set up a task that involves image analysis
> 3. Run the crew with an image URL
> 
> 
> 
> \### Expected behavior
> 
> According to the OpenAI API documentation, the request should be formatted to properly handle multimodal inputs. The image URL should be sent as part of a structured content array with proper type specifications.
> 
> Expected request format (from OpenAI API documentation):
> \`\`\`json
> {
> "model": "gpt-4o",
> "input": \[
> {
> "role": "user",
> "content": \[
> {"type": "input\_text", "text": "what is in this image?"},
> {
> "type": "input\_image",
> "image\_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
> }
> \]
> }
> \]
> }
> \`\`\`
> 
> \### Screenshots/Code snippets
> 
> \## Test Code
> \`\`\`python
> from crewai import Agent, Task, Crew, LLM
> from dotenv import load\_dotenv
> import logging
> 
> \# Configure logging
> logging.basicConfig(
> level=logging.DEBUG,
> format='%(asctime)s - %(levelname)s - %(message)s'
> )
> logger = logging.getLogger(\_\_name\_\_)
> 
> \# Load environment variables
> load\_dotenv()
> 
> \# Define variables
> IMAGE\_URL = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
> 
> \# Configure LLM using CrewAI's configuration
> llm = LLM(
> model="openrouter/openai/gpt-4o"
> )
> 
> \# Create a multimodal agent for image analysis
> image\_analyst = Agent(
> role="Image Analyst",
> goal="Analyze and describe image contents",
> backstory="Expert in image analysis and description",
> multimodal=True,
> llm=llm
> )
> 
> \# Create a task for image analysis with simple prompt
> task = Task(
> description=f"what's in the image? {IMAGE\_URL}",
> expected\_output="A description of what's in the image",
> agent=image\_analyst
> )
> 
> \# Create and run the crew
> crew = Crew(
> agents=\[image\_analyst\],
> tasks=\[task\]
> )
> 
> result = crew.kickoff()
> print(result)
> \`\`\`
> 
> \### Operating System
> 
> macOS Sonoma
> 
> \### Python Version
> 
> 3.12
> 
> \### crewAI Version
> 
> 0.114.0
> 
> \### crewAI Tools Version
> 
> 0.40.1
> 
> \### Virtual Environment
> 
> Venv
> 
> \### Evidence
> 
> \## Actual Behavior
> The request sent to the LLM does not properly format the image data. From the logs, we can see that the image URL is being sent as plain text in the message content rather than being properly structured as a multimodal input.
> 
> Logs excerpt:
> \`\`\`
> 2025-04-10 18:38:56,411 - DEBUG - POST Request Sent from LiteLLM:
> curl -X POST \\
> https://openrouter.ai/api/v1/chat/completions \\
> \-H 'HTTP-Referer: \*\*\*\*\*' -H 'X-Title: \*\*\*\*\*' -H 'Authorization: Bearer sk-or-v1-aaa075b9de083684530d\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*' \\
> \-d '{'model': 'openai/gpt-4o', 'messages': \[{'role': 'system', 'content': 'You are Image Analyst. Expert in image analysis and description\\nYour personal goal is: Analyze and describe image contents\\nYou ONLY have access to the following tools, and should NEVER make up tools that are not listed here:\\n\\nTool Name: Add image to content\\nTool Arguments: {\\'image\_url\\': {\\'description\\': \\'The URL or path of the image to add\\', \\'type\\': \\'str\\'}, \\'action\\': {\\'description\\': \\'Optional context or question about the image\\', \\'type\\': \\'Union\[str, NoneType\]\\'}}\\nTool Description: See image to understand its content, you can optionally ask a question about the image\\n\\nIMPORTANT: Use the following format in your response:\\n\\n\`\`\`\\nThought: you should always think about what to do\\nAction: the action to take, only one name of \[Add image to content\], just the name, exactly as it\\'s written.\\nAction Input: the input to the action, just a simple JSON object, enclosed in curly braces, using " to wrap keys and values.\\nObservation: the result of the action\\n\`\`\`\\n\\nOnce all necessary information is gathered, return the following format:\\n\\n\`\`\`\\nThought: I now know the final answer\\nFinal Answer: the final answer to the original input question\\n\`\`\`'}, {'role': 'user', 'content': "\\nCurrent Task: what's in the image? https://cdn-picture.jingdaka.com/backend\_pic/dst/poster/6ob/2025/03/24/3d6fdf18-aa8d-b7a9-a121-83691464e955.jpg\\n\\nThis is the expected criteria for your final answer: A description of what's in the image\\nyou MUST return the actual complete content as the final answer, not a summary.\\n\\nBegin! This is VERY important to you, use the tools available and give your best Final Answer, your job depends on it!\\n\\nThought:"}, {'role': 'assistant', 'content': "{'role': 'user', 'content': \[{'type': 'text', 'text': 'Describe the contents of the image'}, {'type': 'image\_url', 'image\_url': {'url': 'https://cdn-picture.jingdaka.com/backend\_pic/dst/poster/6ob/2025/03/24/3d6fdf18-aa8d-b7a9-a121-83691464e955.jpg'}}\]}"}, {'role': 'assistant', 'content': 'To analyze and describe the content of the image provided, I will use the image analysis tool.\\n\\nAction: Add image to content\\nAction Input: {"image\_url": "https://cdn-picture.jingdaka.com/backend\_pic/dst/poster/6ob/2025/03/24/3d6fdf18-aa8d-b7a9-a121-83691464e955.jpg", "action": "Describe the contents of the image"}'}\], 'stop': \['\\nObservation:'\], 'stream': False}'
> \`\`\`
> 
> 
> \### Possible Solution
> 
> none
> 
> \### Additional context
> 
> \- The issue appears to be in how CrewAI formats the request to the LLM
> \- The current implementation treats the image URL as plain text rather than properly structuring it as a multimodal input
> \- This prevents the LLM from properly processing and analyzing the image content
> 
> \## Suggested Fix
> The CrewAI library should be updated to properly format multimodal requests according to the OpenAI API specification. The request formatting should be modified to:
> 1. Structure the content as an array of different content types
> 2. Properly specify the image URL as an \`input\_image\` type
> 3. Include the appropriate content type headers and request structure
> 
> Would appreciate any guidance or updates on when this functionality might be properly implemented.

---

<div class="post-metadata">

**Author:** ![like](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/like/32/3895_2.png) [@like](https://community.crewai.com/u/like)\
**Post date:** [June 10, 2025, 1:58pm UTC](https://community.crewai.com/t/crewai-multimodal/1643/14 "2025-06-10T13:58:57Z")

</div>

Hello, have you solved it successfully? I am currently working on a project on video content understanding and need to convert multiple video frames into base64 format for sending! 😀