I’m developing an AI API gateway for multi-agent applications built with CrewAI.
The gateway would provide a common OpenAI-compatible interface and handle model routing, customer-level limits, token usage tracking, retries, and fallback policies.
Multi-agent workloads raise several questions:
- Should quotas be applied per user, crew, task, or agent?
- How should tool-call retries affect usage accounting?
- How can a failed model call be retried without duplicating side effects?
- When should the gateway switch providers?
- Which metadata is useful when debugging a failed crew execution?
I would appreciate practical feedback from developers running CrewAI systems beyond local experiments.
Disclosure: I’m building the gateway discussed here and am interested in architecture feedback.