Agent eval tools?

General
1

Hiya all,

I’m curious how people here are evaluating their CrewAI agents once things start getting a bit more complex?

I mean stuff like:

  • output quality
  • if tool usage is actually improving
  • regressions after changes
  • comparing runs over time
  • sharing results that is easy to understand to other peeps

I tried test.qlankr.com, but I’m wondering what other tools or approaches people here are using?

Are you mostly doing this manually, building your own eval setup or using some other tools?

Would love to hear what people do

2

Hey! I had the same questions, so I built a minimal quickstart that maps each of your questions to a concrete promptfoo pattern:

Your question How this quickstart addresses it
Output quality is-json + JavaScript assertions + llm-rubric for quality grading
Tool usage improvement Agent reports tools_used in structured output; assertions verify specific tools were called
Regressions after changes Saved test cases + promptfoo eval re-runs = automatic regression detection
Comparing runs over time promptfoo view web viewer + JSON/YAML result exports
Sharing results Exported reports or viewer screenshots

Repo: ianTPE/crewai-promptfoo-eval-quickstart

It’s a single-agent CrewAI crew that triages support tickets — nothing fancy, just enough to demonstrate each eval pattern end-to-end. Clone, add your OPENAI_API_KEY, run promptfoo eval.

Complementary to crewai test — that one’s great for overall health; promptfoo is for specific output contracts and regression guarding.

This is a reference quickstart, not a maintained library. PRs welcome if something breaks on newer versions. Hope it helps!