I’ve been working on ReliaMesh, an open-source reliability monitoring tool for AI agents.
One problem I’m trying to address is that successful API calls don’t necessarily mean successful agent executions.
An agent can return HTTP 200 while producing malformed outputs, failing validation, or experiencing regressions after model or prompt updates.
ReliaMesh takes a different approach from traditional tracing: applications explicitly report execution outcomes, and a deterministic detection engine identifies changes in failure rates, latency, token consumption, and retries.
What’s available:
-
Python SDK and self-hosted server
-
Regression detection and incident recovery tracking
-
Model/prompt version tracking
-
No collection of prompts or model responses
-
Reproducible synthetic regression demo
It’s Apache-2.0 licensed and free to self-host.
There isn’t a native CrewAI integration yet, but I’m interested in hearing from developers running CrewAI workflows in production.
How are you currently measuring actual task success and detecting regressions beyond standard tracing?
Feedback and suggestions are welcome.