Building a fully local LLM voice assistant to control my smart home (johnthenerd.com)

Article summary

The author built a fully local LLM voice assistant to control their smart home, using a Protectli Vault VP2420, a managed switch, and two RTX 4060Ti's. The assistant is based on the Mistral AI Mixtral model and uses vLLM for inference. The author had to modify the chat template and fork a custom integration to add support for executing Home Assistant services as JSON. The assistant is designed to be sassy and sarcastic, like the Portal character GlaDOS.

Main themes

• Local LLM voice control
• Smart home automation
• LLM reliability and safety
• Cloud services vs local control
• Wake words and continuous listening
• LLM models and alternatives
• Home automation interfaces
• Privacy and security concerns

What commenters say

• Some commenters are working on similar projects, using different models and approaches to achieve local LLM voice control.
• There are concerns about the reliability and safety of using LLMs for home automation, with some arguing that they can be dangerous if not properly handled.
• Others argue that the risks can be mitigated with proper design and output handling, and that LLMs can be a revolutionary interface for voice commands.
• The use of cloud services like OpenAI's API is seen as a trade-off between ease of use and privacy, with some preferring to run everything locally.
• The importance of wake words and continuous listening is discussed, with some suggesting that a webcam-based approach could be used to detect when the user is looking at the device.
• The potential for LLMs to misinterpret or hallucinate commands is seen as a significant risk, with some arguing that they should not be used for critical systems.
• The need for more recent and commercial-use-permitted models is highlighted, with some commenters suggesting alternatives like bakllava.
• The discussion touches on the idea that LLMs can be used to create more natural and intuitive interfaces for home automation, but also requires careful consideration of the potential risks and limitations.