How do you run a neural network on an ESP32-S3 microcontroller?

You run a neural network on an ESP32-S3 microcontroller by keeping the model tiny, converting it to full 8-bit integer math with TensorFlow Lite, and letting Espressif's ESP-NN kernels use the chip's vector instructions, which can turn seconds of inference into milliseconds or microseconds.

The short answer: small model, integer math, optimized kernels

To run a neural network on an ESP32-S3 microcontroller you need three things: a model small enough to fit in kilobytes, a conversion to full 8-bit integer arithmetic, and a runtime such as TensorFlow Lite Micro backed by kernels written for the chip. Each step attacks a different limit of microcontrollers: tiny memory, slow or missing floating-point hardware, and a modest clock.

A 2026 open-access paper in Applied Sciences shows how far this can go. Its authors built a drilling-rate prediction network with only 280 trainable parameters, quantized it to 5.57 KB, and report an inference time of about 156 microseconds on an ESP32-S3 using TensorFlow Lite Micro.

Why microcontrollers make deep learning hard

A widely cited 2021 study of deploying deep networks on microcontrollers notes that these devices often have less than 1 MiB of memory, and that without a floating-point unit every floating-point instruction must be emulated in software, which adds a large overhead. Many microcontrollers also lack SIMD instructions, so they process one value at a time.

That is why model size comes first. The drilling paper starts from the observation that typical rate-of-penetration models have millions of parameters and run on surface computers, while the goal was to close the predict-and-optimize loop downhole where telemetry is slow. Its team used a genetic-algorithm architecture search and a supervised contrastive training step that constrains the model's internal embedding space, so that accuracy holds up at a very small parameter count.

How 8-bit integer quantization works

TensorFlow's post-training quantization converts a trained float model into one that stores numbers as 8-bit integers. Each tensor gets a scale and a zero point, and the real value is recovered as (integer value minus zero point) times scale. TensorFlow's documentation lists full integer quantization as giving a model about a quarter of the original size and more than three times faster on CPUs, and says a fully integer model is the common case for TensorFlow Lite for Microcontrollers.

Weights can be quantized directly, but activations change with the input, so full integer conversion needs a representative dataset; the documentation suggests a small subset of roughly 100 to 500 training or validation samples to calibrate the ranges. Setting the input and output types to int8 removes the last float operations, so the model can run on integer-only hardware.

The accuracy cost is usually small but not zero. The 2021 study found that 16-bit fixed point matched float32 in all its tests, while 8-bit versions lost up to about 1.1 percentage points on the datasets it tried and sometimes needed quantization-aware training. TensorFlow likewise warns that small networks can lose more accuracy and should be re-checked after conversion.

What the ESP32-S3 adds: vector instructions and ESP-NN

Espressif's ESP-NN library supplies hand-optimized neural network kernels, and on the ESP32-S3 they are assembly versions written to use the chip's vector instructions. It covers the common building blocks: convolution, depthwise convolution, pooling, fully connected layers, elementwise add and multiply, ReLU6 and softmax. It plugs into TensorFlow Lite Micro and can spread heavier kernels across both cores.

The gains are large. Espressif's own benchmarks for int8 models on an ESP32-S3 at 240 MHz list a person-detection model dropping from 2300 ms to 54 ms per inference with ESP-NN, and a MobileNetV3 Small model from 26679 ms to 729 ms. Individual kernels speed up by factors from about 4 to 14.

What is still hard

The tooling is mature, but the limits are real. Only quantizable operations convert, and the converter stops with an error when it meets one that is not. Image models still take tens to hundreds of milliseconds even when accelerated, so real-time vision needs careful model choice. And tiny models need honest validation: the drilling paper reports its optimization gains as predictions from its surrogate model rather than field measurements, and leaves transfer to other downhole tasks as future work.

For hobby and field projects the practical recipe holds: design the smallest model that does the job, calibrate full int8 quantization on real data, measure accuracy again after conversion, and build with ESP-NN enabled.