Browser voice agent lab
Voice AI under hardware constraints
Can a voice agent that listens, thinks and speaks run (almost) entirely inside a browser tab, fast enough to hold a conversation? Each question below is a hypothesis with a working build, measurements and a verdict.
Hypotheses
Can a single HTML file, using existing browser ML libraries and the browser cache, make a voice agent with low latency?
Yes on a GPU: 1.4–2.1 s from the end of your speech to the agent's first sound, on a laptop with a GTX 1050 notebook GPU. Without WebGPU it still works, but takes 12–18 s.
Can we write the inference engine ourselves and still run it all in one HTML file?
The language model now runs on an engine written for this project: hand-written WebGPU kernels (98 tokens/s) and C compiled to WebAssembly with SIMD and threads (28–33 tokens/s), 6× faster than the libraries on the same model and token-for-token identical to Hugging Face. With it, the voice agent answers in about 1 s on a laptop CPU alone, where Hypo 1 needed 12–18 s. Experimental: measured on one laptop.
Learn as Feynman would explain it
- 01The big pictureWhat are the pieces and how do they connect?
- 02ListeningHow does it know you stopped talking?
- 03Speech to textHow does sound become words?
- 04The language modelHow does it write the reply?
- 05Text to speechHow do words become a voice?
- 06The browser as a computerWebAssembly, WebGPU, threads, caches
- 07Latency tricksWhy ~2 s instead of ~5 s
- 08GPU kernelsWhat Hypo 2 is building
- H2Hypo 2 wikiOur own engine: GPU kernels, C/WebAssembly, threads
Source
Everything is on GitHub under the MIT license, including the scripts that produced every number. The models are downloaded from Hugging Face and keep their own licenses (see the README).