Fact Checked by 
The AI startup ElevenLabs launched its new Conversational AI platform on Tuesday (December 3) which allows people to build customizable and interactive voice agents.
On its product page, the company writes, “Add voice to your agents on web, mobile, or telephony in minutes…” It provides several examples, including being used as a support agent, trainer, and concierge.
ElevenLab’s new AI has been made to handle turn-taking and interruptions using a real-time model to predict when a speaker is finished, suggesting it could have a place in the corporate world.
The agents can be created in 31 different languages, with the aim being for the AI to speak to customers in their native language.
On the product page, ElevenLabs has listed ‘Customer Support’ as one of the first use cases.
They say the tool can “handle a wide range of customer inquiries 24/7, reducing wait times and improving customer satisfaction. Agents can troubleshoot issues, process returns, and even upsell products, all while maintaining a consistent brand voice.”
The company states the Conversational AI “can create outbound sales dialers, scheduling agents, interactive game characters, tutors, customer support agents, and more.”
The platform includes features that help users build more interactive agents, including: “Native Twilio integration for handling calls. Server-side and client-side tool calling for added flexibility. Dynamic prompting to create personalized conversations.”
‘Turn-taking and interruption handling hardest to perfect’
It can also connect to a Large Language Model (LLM) of the person’s choosing, including Claude, GPT, Gemini models, or a custom LLM with a server integration.
The lead developer of the project, Jozef Marko, took to X to share more about why the company built it: “We created Conversational AI because our customers wanted to use our Text to Speech API to create interactive agents but found it was challenging to connect Speech to Text, an LLM, and Text to Speech, and even harder to get the interruption handling and turn taking to feel natural.”
We created Conversational AI because our customers wanted to use our Text to Speech API to create interactive agents but found it was challenging to connect Speech to Text, an LLM, and Text to Speech, and even harder to get the interruption handling and turn taking to feel…
— Jozef Marko (@Marko_Jozef) December 3, 2024
He explained how the turn taking and interruption handling was the hardest challenge to crack. “To solve this, we created a real time model of the likelihood that someone is done talking at any moment.
If our agent starts talking and the caller continues to speak over them, we have to handle that gracefully.”
Featured Image: Via ElevenLabs blog
