Technical feedback requested: testing commit validity across Crews and Flows

Crews
, , ,
1

I am exploring a CrewAI integration for CAV-Bench, an open-source deterministic benchmark for evaluating whether an agent’s action remains valid when it commits.

Most agent evaluations focus on whether the task was completed or the expected final state was reached. CAV-Bench looks at failure modes that can remain hidden behind that apparent success, including:

  • state changing between planning and execution;
  • duplicate side effects after an ambiguous retry;
  • authority changing before an action commits;
  • partial execution across multiple systems;
  • incomplete compensation or escalation;
  • an agent reporting success when the committed state does not support it.

For an initial CrewAI implementation, I am considering a controlled scenario built around Crews and Flows:

  1. One agent reads operational state and recommends an action.
  2. The test harness changes the underlying state or authorization.
  3. A second agent or Flow step attempts the external action.
  4. The evaluator checks whether the action was blocked, committed once, reconciled, compensated, or escalated.

The benchmark evaluates each run across five dimensions:

  • intent;
  • authority;
  • state;
  • execution;
  • recovery.

Before building the adapter, I would appreciate feedback from the CrewAI community on two questions:

  1. Are CrewAI Flows the right layer for implementing deterministic state changes, retries, and recovery paths?
  2. Which state, event, persistence, or checkpoint interfaces would provide the most reliable evidence that an external action was attempted, committed, retried, compensated, or escalated?

I am considering starting with four scenarios covering stale state, duplicate tool execution, ambiguous retries, and partial workflow failure.

This would be an independent community integration, not an official CrewAI benchmark. I am looking for technical guidance before implementing it, rather than endorsement or formal adoption.

Repository: GitHub - Harimay23/cav-bench: Benchmarking commit-time validity for tool-using AI agents. · GitHub
Version DOI: CAV-Bench: Commit-Time Action Validity Benchmark | Zenodo