Kingston Kuan

Voice AI in 2026

Unknown — Sun, 22 Mar 2026 00:00:00 +0000

Since I’ve been exploring the extended universe of voice AI and voice agents, I’ve met a lot of people working with them in the real world and it’s not just the average customer support bot anymore. I’ve met teams building debt collection agents, emergency services routing, and even language-specific services for regions outside the US.

In the spirit of Learning in Public</a>, I wanted to write down some of what I know in case someone might learn from it.

I mostly cover conversational voice agents using the full pipeline in this post (STT -> LLM -> TTS) but they are far from the only use case of voice AI. Voice interfaces like Wispr Flow</a> or Willow</a> use STT and LLM without TTS for AI-powered voice dictation in any app. I also maintain an open source voice interface called Tambourine</a>. Bulk transcription and summarization tools like Granola</a> have less real-time requirements like latency and can optimize for intelligence. You could even go the other direction and use only TTS for generating media content. The voice stack is flexible so you can pick the pieces that you need.

I focus on conversational voice agents just because they touch the entire stack, making them a good lens for understanding the technologies involved.

So, if you’re wondering if it’s possible to run voice agents in production in 2026, for me the answer is yes. Even though it might not be as easy as building web apps just yet, many players in the industry are vying to make it simpler for anyone to build in voice. If you’re interested in building a voice agent today, we can start off with choosing what layer of abstraction to build upon.

This post names a few models and providers as examples. There are many more out there so do your own research.

Layers of abstraction</h2>
High-Level: Services like Vapi</a> & Retell</a> offer orchestration-as-a-service. They handle the handoffs between STT, LLM, and TTS, telephony bridging, and complex state management of a live conversation. This path is often chosen for faster speed to market.</li>
Low-Level: LiveKit</a> & Pipecat</a> are open source frameworks that give you full control over the agent pipeline. Instead of calling a black-box API, you’re building the “brain” of the agent yourself. This path is typically used if you require granular control or need to optimize costs.</li> </ol>
Using higher level orchestration services is more straightforward so let’s focus on the low level path to learn more about the underlying technology. When you own the pipeline you need to start thinking about the two fundamental technical challenges of voice AI: The Network and The Model Pipeline.
For a deeper technical dive into these components, the Voice AI and Voice Agents Primer</a> is an excellent resource. </blockquote>
The Network</h2>
The average developer rarely thinks about networking problems because most web apps are built around request-response patterns. That works fine because we’ve spent years optimizing the internet for that. When it comes to building voice AI on top of streaming protocols, optimizing the network becomes crucial for performance and quality. You can have the fastest and smartest LLM in the world, but if your audio packets are getting lost hopping across the globe, the conversation will feel broken.
When dealing with audio streaming for voice, you will hear a lot of different infra terms that do not come up as often in the classic HTTP and REST API stack: Websockets, WebRTC, SFU, SIP, PSTN, and so on.
As much of this still requires physical infrastructure, you will likely lean on existing global edge networks. Even LiveKit</a> and Daily</a> (the company behind Pipecat) provide cloud solutions to abstract this away. Daily and LiveKit grew significantly during the teleconferencing era and both have since applied their expertise to building for voice agents. Their global networks are hard for new entrants to replicate quickly, which makes them a good foundation for voice AI infrastructure.
That being said, I’ve been surprised by how many teams are starting to roll their own regional network stack. If you’re building for a specific market (say, Vietnam), you don’t necessarily need a global mesh. You can stand up your own WebRTC infra, using LiveKit’s open-source stack</a> for example. For regional users, this approach can be more efficient than relying on generalized cloud solutions.
Telephony is the final piece of this puzzle. You still need CPaaS (Communications Platform As A Service) providers like Twilio</a> or Telnyx</a>.
Twilio is an established market leader in CPaaS with a mature ecosystem that many enterprises have historically relied on (I used it back in 2019) while Telnyx has focused on vertical integration specific to voice AI, owning their own networking and colocating GPUs for fast model inference.
The Model Pipeline</h2>
With the underlying network out of the way, you still have to worry about the actual intelligence part of the agent. Today, the industry standard is still the 3-model pipeline: STT (Speech-to-Text), also called ASR (Automatic Speech Recognition) -> LLM (Large Language Model) -> TTS (Text-to-Speech).
You might wonder why we aren’t all using direct Speech-to-Speech (S2S) models like the Gemini Live API. While such models might be more capable of capturing nuanced emotional cues, production agents still face two major hurdles:

Intelligence vs. Latency: The models currently capable of handling complex tool calling with the highest reliability are still text-based.</li>
Cost: Running a conversation through S2S is still significantly more expensive than multi-model pipeline.</li> </ol> By owning the pipeline, you can optimize for latency, cost, and advanced features. Developers typically aim to hit an 800ms voice-to-voice round trip</a>.
STT / ASR</h3>
Providers like Deepgram offer sub-300ms transcription</a>, while Speechmatics provides features like diarization for multi-user conversations</a>. You can also often self-host STT models as they are smaller and don’t require GPU clusters, like Nemotron ASR which I use in my personal setup</a> or optimized versions of OpenAI’s Whisper. This isn’t an option with a monolithic S2S model.
Pipecat’s recent STT benchmarks</a> are a great reference for anyone building in this space.
LLM</h3>
LLM selection for voice agents follows similar principles as other agentic use cases. You need reliable tool calling, instruction following, and reasoning, but with the added focus on latency. Heavier models give better reliability but add round-trip time, so many production agents use lighter models like Gemini Flash. Providers like Cerebras</a> specialize in delivering fast model inference for open weights models like gpt-oss-120b.
TTS</h3>
For the final leg, TTS converts the LLM’s text response into speech. Cartesia</a> and ElevenLabs</a> are some common providers you might have heard of that offer realistic, low-latency speech generation across many languages and tones. The TTS space hasn’t quite settled on a dominant approach yet, with different model architectures still competing. For example, Cartesia’s Sonic is built on State Space Models (SSMs) optimized for streaming, while ElevenLabs uses transformer-based architectures. It’s an area worth keeping an eye on as the research matures.
Collapsing STT and LLM</h3>
An interesting approach that blurs the line between the 3-model pipeline and full S2S is Ultravox</a>. Instead of transcribing audio to text and then feeding it to an LLM, Ultravox encodes audio directly into the LLM’s token embedding space. This effectively merges the STT and LLM steps into one, reducing latency while still outputting text that can be fed to a standard TTS model. A tradeoff here is inspectability. With the intermediate transcript from STT you can log, debug, and audit, but with this this approach that visibility is lost.
Taking Turns</h2>
Even with a perfect network and fast models, a voice agent can still feel robotic if it doesn’t know when to jump into a conversation or stop talking. Turn management is where some of the hardest engineering challenges are being tackeled right now, solving for VAD and Interruptibility for example.

VAD (Voice Activity Detection): How do you know when someone is finished talking? Naive volume-based VAD fails because natural speech has pauses between clauses, and silence alone doesn’t mean the speaker is done. We’re moving toward semantic VAD, using a tiny model to understand the intent of the pause. By far the most popular solution right now is Silero VAD</a>.</li>
Interruptibility: If the agent is speaking and the user starts answering a question, how does the agent handle it? It can’t keep talking over the user and has to stop appropriately to take in the new information quickly and modify its response. Some providers like Pipecat provide their own Smart Turn</a> model to try and solve for this.</li> </ul>
Observability and Evals</h2>
Another piece of the puzzle in the voice space is observability and evals. This is common in other agentic applications too but the voice stack comes with additional challenges.
Voice agents are harder to evaluate than text agents because they have more dimensions of failure. It’s not just “did it give the right answer?” but “did it interrupt the user inappropriately?” or “was the latency spike caused by the model provider or an intermittent network issue?”
A common approach today is to focus on binary ratings for specific behaviors (“Did the agent complete the booking?” or “Did the agent correctly verify the caller’s identity?”) and keep a human in the loop to review traces, rather than looking for a single accuracy score based on arbitrary metrics. You can curate eval sets from real conversations, often called “Golden Datasets”, to test how tweaking the model pipeline would affect past conversations. Startups like Hamming AI</a> or Coval</a> are attempting to automate parts of this.
It’s time to build</h2>
Building in voice right now feels like the early days of LLMs. Whether you rely on out-of-the-box solutions or deploy your own infra, there hasn’t been a better time to be building in voice. There is plenty more to say on any of these topics and I’m barely scratching the surface. While I’m still learning and growing in the space myself, I hope you learned something new from this post, or at least have more resources to dive deeper.
Acknowledgements</h2>
I would like to thank all the great folks I’ve met in the community that have advised and supported me with my learning. Kwindla</a> from Daily, Edgars</a> and the rest of the Speechmatics</a> team for the support in building Tambourine</a>. Travis</a> for encouragement on writing this. And everyone else I have met building in the voice community!

Dictating anywhere with NVIDIA open models - Nemotron ASR + Tambourine

Unknown — Tue, 03 Feb 2026 00:00:00 +0000

I saw NVIDIA’s announcement</a> on X about their new series of open Nemotron models and ended up reading a blog post from Daily.co</a> about building voice agents with Nemotron and Pipecat.
I’ve been using voice dictation daily since building Tambourine</a>, so I decided to test Nemotron with it. It’s fast for real-time use, and the accuracy is solid, unfortunately you do need a CUDA GPU. Here’s how I set it up.
My Setup</h2>
Tambourine is the desktop app that ties everything together. It captures audio, sends it through the pipeline, and types the result wherever my cursor is.
[Microphone] -> [Nemotron ASR] -> [LLM] -> [Insert text at cursor]</code></pre> There are great options for AI voice dictation like Wispr Flow or Willow, but Tambourine is open source and allows you to have full control over the pipeline: swap STT/ASR and LLM, customizable formatting prompts, and import dictionaries for different domains. As Tambourine is built on top of Pipecat, adding Nemotron as an ASR provider was straightforward. I’m running this on a desktop with an RTX 4080 Super. Nemotron ASR requires a GPU with CUDA support, which means Windows or Linux only. macOS users can still use Tambourine with cloud STT/ASR providers.You can use any LLM provider compatible with Pipecat, I tend to use Cerebras</a> with gpt-oss-120b</code> for fast cloud inference. Setting Up Nemotron ASR</h2> To run only the ASR model via Docker, I forked PipeCat’s example</a> to strip it down to an ASR-only image</a>. git clone https://github.com/kstonekuan/nemotron-january-2026 cd nemotron-january-2026 docker build -f Dockerfile.asr -t nemotron-asr . docker run --gpus all -p 9765:9765 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ nemotron-asr</code></pre> Once running, you can check the logs in docker. Setting Up Tambourine</h2> See the Tambourine repo</a> for full setup instructions. To use Nemotron, set NEMOTRON_ASR_URL</code> in .env</code> and select Nemotron as the STT provider in the app settings. NEMOTRON_ASR_URL=ws://localhost:9765/</code></pre> If running the Tambourine server in docker as well you might have to use NEMOTRON_ASR_URL=ws://host.docker.internal:9765/</code></pre>Wrap Up</h2> I’ve really enjoyed exploring this stack, and look forward to trying out new models as they come out. AI-powered voice dictation has changed how I interact with all my apps. Especially when it comes to using AI tools like Claude Code or ChatGPT, I find myself reaching for the keyboard less and just thinking out loud, trusting Tambourine to capture my intent on top of the words. Quick Links: Tambourine</a></li> Pipecat Nemotron Example Fork</a></li> NVIDIA Announcement</a></li> Daily.co Blog Post</a></li> PipeCat</a></li> Cerebras</a></li> </ul>
Why is it so hard to get rid of deepfakes? Unknown — Sat, 07 May 2022 00:00:00 +0000 Originally published on Medium</a>. On April 17, 2018, the world was greeted by Former President Barack Obama saying outrageous things in a video titled “You Won’t Believe What Obama Says In This Video!”. About 30 seconds in, it is revealed that is a deepfake allowing the famous comedian Jordan Peele to use the face and voice of “Obama”. As technology advances further, higher forms of media manipulation such as video and sound have been made possible and must be addressed quickly. </iframe> </div> Deepfake technology</a> can produce fake content indistinguishable from real content by the human eye. This technology has risen in popularity due to abuse in sensational media by the spread of misinformation and infringement of privacy. However, not only is it impossible to eradicate these misuses completely but the core technology also has other benefits and thus does not warrant an outright ban of its applications. Furthermore, regardless of the benefits or detriments, the advent of deepfake technology means media regulations must keep up before widespread use is commonplace. While technology is being developed to definitively identify Deepfakes</a>, new regulations</a> should be put in place to cater to their uses. Specialized categories should be set up on platforms to ensure users comply with the rules and technical agencies must be set up for appropriate policing. Manual manipulation of images and videos is neither new nor rare and the war on misinformation has been fought on many fronts. Ranging from digital editing in marketing to false evidence in court, a big hurdle is being faced now with deepfakes being created by artificial intelligence. Yet, it is believed that the very technology capable of detecting deepfakes</a> can also be used to improve them and thus even harder to detect. In a battle of computing power and datasets, it is imperative to not only support the detection but be able to keep this away from nefarious users. Yet the development of such powerful technology has traditionally been open source and only possible with the collaboration of many individuals made possible by the interconnectivity of the web. It is not reasonable to rely on such teams to be able to securely store their algorithms. It is better to instead consider a hybrid approach to development where the project can be managed by trusted parties such as a government body or large private institution while still allowing for open contributions. Technology evolves rapidly and where public regulations often fail to keep up, fast-moving private bodies are well-suited for this role. Given that the open-source nature of artificial intelligence development is unlikely to change significantly, regulations prove even more crucial than combative technology. Media platforms, such as Youtube or Facebook, are often in the most appropriate position to regulate deepfake technology</a>. Not only do they have access to large datasets, but they have already been dealing with similar hostile uses of their platforms and have amassed many skilled employees. The power given to these privately owned platforms by the internet has already been immense, being able to affect economics, politics, and societies on a large scale. Indeed, with this should come the responsibility to police their own platforms and be punished for not investing heavily in countermeasures. Current measures to combat deepfakes have been to put a lid on them rather than attempts at integration. Yet, completely erasing deepfakes is a fruitless endeavor. It would more productive to embrace deepfakes with their own category</a> on these platforms for creative uses. In this manner, the technology may still develop while being healthily regulated. By complying with ethical and legal standards, personal privacy and commercial rights can be protected more smoothly. Desensitisation will also play an essential role in aiding the public with understanding deepfake technology which helps them to identify and report illegal activities. With the stigma behind deepfakes shifted away from sensational topics such as politics and obscene material, the research behind combative technologies also gains an edge in terms of financial and technical support. Existing agencies still play important roles</a> in attempting to regulate deepfakes. However, their current capabilities are highly restricted. A new agency would be more suited to dealing with the impending threat of deepfakes. This agency must be technically equipped to deal with deepfakes and must also be able to administer appropriate penalties to rule breakers. As the internet has no borders, an international agency, like what the World Health Organisation is to healthcare, needs to create general guidelines that can thereafter be tailored to individual countries’ needs. This body should punish individual agencies for not being able to properly police within their jurisdiction. On the other hand, it is worth analyzing the deepfake situation from a social perspective. We need to understand the reasons behind the creation of deepfakes and their uses</a>. As advanced as technology gets, even though the intelligence being used is artificial, there is always a human aspect to how and when it is used. By focusing on the community and their technological literacy, malicious use is reduced, and misconceptions corrected. As a remedy, these measures are hard to quantify but similarly, it is known that better education contributes to a lower crime rate</a>. Targeting the root behind the misuse of deepfakes also allows greater freedom in its research which has wider benefits than just entertainment. In conclusion, deepfakes are very much a part of modern technology and therefore difficult to eradicate. Governments and platforms have important roles to play in contributing to the development of technology capable of identifying deepfakes. As fighting solely on the technical front is insufficient, laws and regulations must attempt to pull ahead by catering to expected growth in deepfakes with deliberate exposure, the appointment of responsibility, and education. If left unchecked, deepfakes may begin to rule all media and the distortion of reality may become the rule rather than the exception. Building a chatbot for Angel & Mortal Unknown — Sun, 13 Jun 2021 00:00:00 +0000 Originally published on Medium</a>. I was starting an hour long commute home when my phone came abuzz. A friend was organizing the annual Angel & Mortal game for our residential college community and was interested in leveraging some online platform for anonymous communication between players. The rules of Angel & Mortal are simple: Each player is assigned a Mortal to whom they act as an Angel.</li> The Angel must give gifts to or play pranks on their Mortal without being discovered.</li> </ul> Typically, players would send messages to the organizers to be manually forwarded to their Mortal or Angel, asking for their likes/dislikes or informing them of new gifts. I immediately found this project to be simple yet impactful and was compelled to help. Unfortunately the game was starting soon, in a week, so I had to work fast. Overview</h2> This bot was developed in Python using the python-telegram-bot</a> package. This allows you to run a simple server that will poll for messages sent to your bot on telegram. Before starting, you will need to set up a bot using the BotFather and generate a token (instructions here</a>). The initial player data are loaded from CSV. Original code used when I was running the bot (sorry if it’s messy!) can be found here: https://github.com/kstonekuan/angel-mortal-bot</a> Development</h2> Player class</h3> Before diving into the chatbot logic, it’s important to understand the theory of the game itself. Each player has an Angel and a Mortal. Furthermore, your Angel is another player in the game who has you as their Mortal and vice versa for your Mortal. From an OOP perspective, each player can be represented as an object that references other player objects. Along with some telegram attributes, this can be represented by a Player class. class Player: def __init__(self, username, room, angel, mortal): self.username = username self.room = room self.angel = angel self.mortal = mortal self.chat_id = None</code></pre> While not crucial, you might recognise this as a graph node from graph theory. In particular, the game is designed as a directed graph</a>. Each node has two outgoing edges (to the Angel and Mortal) and two incoming edges (from the Angel and Mortal). Loading players</h3> While it might be possible to use our system to generate Angel-Mortal mappings, organizers often have a more in-depth understanding of social dynamics within the group. I chose to load provided mappings from a CSV file, as opposed to using something more complex (but efficient at scale), as it was simpler for organizers to modify the data. After loading, players can be stored in a dictionary or hash map in memory, similar to an adjacency list. def load_players(filename): players = {} with open(filename, 'r') as f: reader = csv.DictReader(f) for row in reader: players[row['username']] = Player( row['username'], row['room'], row['angel'], row['mortal'] ) # Link player objects for player in players.values(): player.angel = players[player.angel] player.mortal = players[player.mortal] return players</code></pre>Forwarding messages</h3> Note: I reference parts of the python-telegram-bot API without explaining them, please refer to their tutorials</a> for this. You might have noticed that our Player class includes a username and chat_id. The username is used to identify players and is provided by the organizers while chat_id is what is used to send messages. We cannot send messages directly to a username as it would be unclear how we want the bot to communicate such as in a private message or group chat. Each user needs to start a chat with the bot first in order to get the chat_id that is generated by Telegram. I also went one step further to reject users that do not appear as a key in the dictionary as they are not registered players in the game. def start(update, context): username = update.effective_user.username if username not in players: update.message.reply_text("You are not registered for this game.") return players[username].chat_id = update.effective_chat.id update.message.reply_text(f"Welcome {username}! You can now send messages.")</code></pre> Conversation handlers</a> can be used to create simple menus to guide users through your bot flow. For this bot the user must first choose either their Angel or Mortal before sending their messages across. They may cancel the conversation at any time to switch between the two. Thanks to the class created in previous steps, it is not difficult to find the reference to their Mortal or Angel and get their respective chat_id to forward messages. def send_to_mortal(update, context): username = update.effective_user.username player = players[username] mortal = player.mortal if mortal.chat_id is None: update.message.reply_text("Your mortal hasn't started the bot yet.") return # Forward the message context.bot.send_message( chat_id=mortal.chat_id, text=f"Message from your Angel:\n\n{update.message.text}" ) update.message.reply_text("Message sent to your mortal!")</code></pre> The result should look something like this (with custom names for Angel and Mortal): Deployment</h3> So far you might have been testing your bot locally but it is probably not a good idea to keep to run this on your own machine for the entire duration of such events. I personally made use of Microsoft Azure free student credits</a> to host but there are many options</a> available. Improvements</h3> As this was built in a few days with little time for testing there are many improvements you may want to make. Common feedback from players was that they could not send pictures or emojis through the bot. I had forgotten to account for this which might have to do with the Filters</a> module.</li> For long term games that require modifications or continuity you may also consider building a proper backend database such as using SQL or NoSQL</a> instead of a simple CSV.</li> Using the Player class and directed graph design, you can develop advanced algorithms to generate, validate or even modify the player graph for larger games.</li> </ol> I really enjoyed building this bot and helping out my community. This is my first article about programming so I hope you enjoyed reading and am always open to feedback about my writing!