Use OpenAI Realtime API over WebSocket as tool

[arnoob]

Hi everyone,

I’m working on Fine Voicing, a tool to generate conversations with Voice AI agents, like OpenAI realtime API or Ultravox.

It’s open source here: GitHub - Fine-Voicing/fine-voicing-crewai: Fine Voicing is a tool to generate test conversations with Voice AI agents

I’m trying to use OpenAI Realtime API as a tool but I’m not quite sure on how-to implement this and the docs isn’t really helping me here.
I have a first preview of the implementation visible in a working branch: fine-voicing-crewai/src/test_runner.py at openai-realtime · Fine-Voicing/fine-voicing-crewai · GitHub

How should I approach writing this tool? Thanks for your help!

[arnoob]

After quite some iterations and struggles, also due to my lack of knowledge of async programming in Python, I have a working version visible in the branch fine-voicing-crewai/src/tools/openai.py at openai-realtime · Fine-Voicing/fine-voicing-crewai · GitHub

I feel it is quite complex, involving a dedicated thread to maintain the websocket connection throughout the generation, but I could not make it work in a simpler way yet.

Hope it’ll help some of you. Also looking forward to hearing feedback from the community!

[rokbenko]

@arnoob Congratulations on solving the issue and your intention to help other developers by sharing the code. That’s very welcome. But the thing is, if you only post the link, it might get dead (e.g., you rename the GitHub repo). Please edit your answer above and add the code too.