What is split inference, and how can several small IoT devices run one neural network?

The short answer: cut the model into segments and pass the results along

Split inference means running one neural network across more than one device. The model is cut at chosen layers into segments. The first device runs the first segment on the raw input and sends its output, an intermediate tensor of activations, to the next device, which runs the next segment, and so on until the last device produces the prediction. Nothing about the math changes; the same layers run in the same order, just on different chips.

For small IoT devices the main reason to do this is memory. A microcontroller has a fixed amount of flash for storing weights and a small amount of RAM for working buffers. A model that does not fit on one board can still run if its layers are spread over several boards, each holding only its own share.

How split computing is normally done between a device and a server

The idea first grew up between a phone or sensor and an edge server. A survey of the field by Matsubara, Levorato and Restuccia describes the options: run the whole model on the device, send the raw input to a server and run it there, or split the model into a head on the device and a tail on the server. Splitting spreads the computing load and sends a smaller representation over the radio instead of the raw input, which can also reveal less about that input.

The catch is that a split only saves bandwidth if the tensor at the cut is smaller than the input. The survey notes that early work searching for the best split point often found that the best choice was the very first or very last layer, because common image models do not shrink their data early. Researchers responded by adding artificial bottlenecks, small encoder layers trained to compress the features at the cut, often with knowledge distillation so accuracy holds up. A related trick, early exiting, adds cheap classifiers partway through so easy inputs can stop early.

Splitting a model across a group of microcontrollers

The same technique can link several microcontrollers with no server at all. A 2025 study in the journal Sensors split a small convolutional network for pump vibration monitoring across three ESP32-S3 boards. The full model, with 56,289 parameters, would not run on one board's internal SRAM: the authors report an 11,600-byte shortfall once system tasks and runtime overhead were counted. Cut at its pooling and flattening points into three sub-models, one per board, it ran with predictions identical to the single-board baseline using external PSRAM.

In that setup each node sent its activation tensor, with its shape, to the next node over TCP on a Wi-Fi network. The authors chose the cut points by hand and list an automatic partitioning algorithm as future work. Their test used one three-node chain.

What is hard about it: failures, memory limits and slow links

A chain of devices is only as reliable as its weakest link. If one board in a split pipeline loses power or drops off the network, its segment is gone and inference stops for everyone. A 2026 arXiv preprint, Replica-Aware Split Inference for Edge IoT Systems by Jenhani, Razmi and Jukan, tackles this by copying the segments of failed devices onto surviving devices as replicas.

Choosing where those replicas go is a balancing act. Each device has limited flash and runtime memory, links between devices run at different speeds, and every extra hop adds transmission delay on top of computation time. The authors frame the choice of replica hosts and communication protocols as a mixed-integer linear program that minimizes end-to-end latency after reconfiguration while respecting the memory limits. They tested it in simulation with MobileNetV2 on mixed IoT networks and on a network of 15 devices with link rates cut by up to 75 percent. They report modeled latency on average 6.8 percent lower than a computation-greedy placement and 18.3 percent lower than random placement.

Taken together, the work points to the practical checklist for anyone trying this: find cut points where the data is small, budget the RAM each segment really needs, measure the links rather than assume them, and plan for what happens when a node disappears.