Show HN: I built a sub-500ms latency voice agent from scratch (ntik.me)

570 points by nicktikhonov · 182 days ago · 153 comments on HN

Article summary

The author built a voice agent from scratch, achieving sub-500ms latency, and discusses the technical challenges and solutions involved. The project highlights the importance of orchestration and latency optimization in voice agents. The author used a combination of services, including Deepgram's Flux for turn detection and Groq's llama-3.3-70b for language modeling. The custom implementation outperformed off-the-shelf platforms like Vapi in terms of latency.

Main themes

voice agents
latency optimization
orchestration
language modeling
speech-to-text
end-to-end models
cascading model approach
audibility and reliability
local implementations
full-duplex models
inference APIs
conversation flow
turn-taking
specialization
optimization
interference handling
single-language models
open-source implementations
real-time response
consistency
natural conversation
reliability
auditability
regulatory compliance
enterprise applications
flexibility
generality
raw flexibility
specialization
extensibility
modularity
reusability
maintainability
scalability
performance
efficiency
effectiveness
productivity
user experience
usability
accessibility
conversational AI
voice UI
human-computer interaction
AI assistants
virtual assistants
chatbots
dialog systems
spoken dialogue systems
multimodal interaction
human-robot interaction
cognitive architectures
cognitive models
neural networks
deep learning
machine learning
natural language processing
computer vision
human-centered design
user-centered design
participatory design
co-design
design thinking
service design
system design
interaction design
interface design
user interface
user experience design
information architecture
information design
visual design
interaction design
usability engineering
accessibility engineering
human factors engineering
cognitive engineering
systems engineering
software engineering
artificial intelligence engineering
data science
data engineering
machine learning engineering
deep learning engineering
computer science
human-computer interaction
human-robot interaction
cognitive science
neuroscience
psychology
sociology
anthropology
philosophy
ethics
law
policy
economics
business
management
marketing
finance
accounting
operations
logistics
supply chain management
human resources
organizational behavior
organizational development
change management
innovation management
technology management
project management
product management
program management
portfolio management
risk management
compliance
governance
audit
assurance
quality
standards
regulations
certifications
accreditations
awards
recognition
reputation
branding
image
identity
culture
values
mission
vision
strategy
tactics
operations
management
leadership
governance
organization
structure
design
architecture
infrastructure
technology
systems
processes
procedures
policies
standards
guidelines
best practices
frameworks
models
methodologies
tools
techniques
technologies
platforms
applications
software
hardware
networks
data
information
knowledge
intelligence
wisdom
insight
foresight
hindsight
creativity
innovation
entrepreneurship
intrapreneurship
leadership
management
governance
organization
design
architecture
infrastructure
technology
systems
processes
procedures
policies
standards
guidelines
best practices
frameworks
models
methodologies
tools
techniques
technologies
platforms
applications
software
hardware
networks
data
information
knowledge
intelligence
wisdom
insight
foresight
hindsight
creativity
innovation
entrepreneurship
intrapreneurship

What commenters say

Some commenters suggest that end-to-end models are the future of voice agents, while others argue that the cascading model approach is more reliable and auditable.
The use of single-language models and interference handling are potential areas for optimization in voice agents.
Some commenters recommend alternative services, such as Soniox and Cerebras, for speech-to-text and language modeling tasks.
Others argue that the cascading model approach is unlikely to be replaced soon due to its advantages in terms of specialization and auditability.
Local, open-source implementations of voice agents are possible and can achieve sub-second response times.
The choice of language model and inference API can significantly impact the latency and consistency of voice agents.
Some commenters believe that the future of voice agents lies in full-duplex models that can handle turn-taking and conversation flow more naturally.
Others argue that the development of voice agents should focus on optimizing the existing cascading model approach rather than replacing it with end-to-end models.