Can an ESP32-P4 microcontroller do offline speech-to-text?

The short answer: yes, with a streaming transducer and very low-bit weights

An ESP32-P4 microcontroller can do offline speech-to-text, writing down arbitrary spoken English as it is spoken, with no cloud and no neural accelerator. A 2026 open-source project and accompanying arXiv preprint, Streaming Speech Recognition on a Microcontroller: a 65.7 M-Parameter Transducer in 2 and 4 Bits on the ESP32-P4, ports a streaming Zipformer2 transducer to the chip and reports a word error rate of 4.34% on the LibriSpeech test-clean set, against 3.94% for the same model on a PC.

That is different from the keyword spotting most microcontrollers do. Command recognizers listen for a fixed list of phrases. A full transcriber has to produce any word in its vocabulary, which normally means a model with tens of millions of parameters and the memory and compute of a phone or a server.

Why streaming speech recognition uses transducers

Google's 2018 paper on streaming end-to-end recognition for mobile devices explains the design choice. A recurrent neural network transducer (RNN-T) maps audio frames directly to characters or word pieces, with an encoder for the audio, a prediction network that remembers what has already been emitted, and a joint network that combines them. Because its outputs never depend on future audio, decoding can start before the speaker finishes. Attention-based encoder-decoder models usually need the whole utterance first.

Transducers also beat the older CTC approach in that paper, because CTC assumes each output label is independent of the ones before it. Google's word-piece RNN-T had about 120 million parameters and reached 6.8% word error rate on voice search versus 9.3% for its CTC baseline, while running about twice as fast as real time on a Pixel phone after quantization.

What Zipformer adds

The encoder used on the ESP32-P4 is a Zipformer, a Transformer-style speech encoder from the open-source icefall project. Instead of processing audio at one fixed frame rate the way Conformer does, Zipformer runs six stacks at different rates, from 50 frames per second down to 6.25 and back up, in a U-Net-like shape. The middle stacks see much shorter sequences, which cuts computation. The Zipformer paper also shares attention weights across modules, replaces LayerNorm with a simpler BiasNorm, and reports more than 50% faster inference than prior encoders with state-of-the-art accuracy on LibriSpeech.

How the model fits into a $5 chip

Memory is the hard limit. The released model is 71 MB; the ESP32-P4 board used has 32 MB of PSRAM and 16 MB of flash, and an ordinary 8-bit version would not fit. The project instead stores each weight matrix at 2 or 4 bits, choosing the width per matrix by how sensitive accuracy is to it, which brings the weights to 26.65 MB with no retraining. Because flash is too small, the weights load from a microSD card in about 2.9 seconds at boot.

Speed comes from integer math and assembly. The ESP32-P4 has two RISC-V cores at 400 MHz and no NPU, but it does have a SIMD extension called PIE that performs 16 int8 multiply-adds per instruction. The project skips existing frameworks such as ESP-DL and TensorFlow Lite Micro and uses a custom C engine with hand-written PIE loops. Its first port needed 1.3 seconds to process 1.28 seconds of audio, slower than real time; the optimized version takes about 0.9 seconds per 1.28-second chunk, including drawing to the screen. Words are finalized about two seconds after they are spoken.

Limits worth knowing before you build on it

The reported accuracy is for clean read speech. On the harder LibriSpeech test-other set the error rate rises to 10.26%, and noisy rooms and strong accents remain difficult. The model is English only. It needs revision v3 of the ESP32-P4X, because older chips use different PIE instruction encodings, and power consumption has not been measured. The project is Apache-2.0 licensed, supports custom hotwords from a text file, and ships offline voice-control demos that need no language model.