How to use the qwen2.5-vl-3b-instruct model with the CrewAi?

LLMs
1

"I am trying to use the Qwen series model with the Hugging Face provider locally, but it does not support CrewAI. I am using vLLM to host the model on an API endpoint, but when I call the model through the LLM class, I encounter an error.

I have tried multiple approaches, including using FastAPI to serve the model with api_base or base_url, but it still doesn’t work. I am using the Qwen model for image-based question answering. Below is the code I used:

python

CopyEdit

llm = LLM(
    model="huggingface/Qwen/Qwen2-VL-2B-Instruct",
    api_base="http://0.0.0.0:8000/v1/chat/completions",
)

Can you help me resolve this issue? Also, does litellm support the Qwen-Vision model? If yes, how can I use the Qwen model with CrewAI?

Thanks in advance!"

2

LiteLLM supports all models on VLLM. VLLM | liteLLM

Try changing to:

llm = LLM(
    model="hosted_vllm/huggingface/Qwen/Qwen2-VL-2B-Instruct",
    api_base="http://0.0.0.0:8000/v1/chat/completions",
)

I think your issue is on the model path or the api_base.

3

Hi @zinyando, I tried using the LLM class, but it didn’t work. Below is my full code. Please suggest how to use the Qwen model. If there is another idea or technique to use the Qwen model, please let me know.

from crewai import Agent, Task, Crew, LLM
llm = LLM(
    model="hosted_vllm/huggingface/Qwen/Qwen2-VL-2B-Instruct",
    api_base="http://0.0.0.0:8000/v1/chat/completions",
)

image_analyst = Agent(
    role="Image Analyst",
    goal="Analyze the given image and provide detailed insights based on the provided question",
    backstory="Expert in visual analysis with deep knowledge of design, composition, objects, patterns, and features. Can accurately describe and interpret images across various contexts.",
    multimodal=True,
    verbose=True,
    llm=llm
)

task = Task(
    description=(
        "Analyze the image located at https://as1.ftcdn.net/v2/jpg/10/64/66/34/1000_F_1064663493_PG2uY9VYvZPVxXvmFwIOvGDSNlfMDeIL.jpg and describe the word written on the image."
    ),
    expected_output="A detailed analysis and response based on the given image and question.",
    agent=image_analyst
)

crew = Crew(
    agents=[image_analyst],
    tasks=[task]
)

result = crew.kickoff()

Error:
litellm.exceptions.NotFoundError: litellm.NotFoundError: NotFoundError: Hosted_vllmException - Error code: 404 - {'detail': 'Not Found'}