What is feed-forward 3D Gaussian splatting, and how does it build a 3D scene without per-scene training?

The short answer: one network pass instead of per-scene optimization

Feed-forward 3D Gaussian splatting is a way to build a 3D Gaussian splat scene by running a trained neural network over a few photos once, rather than fitting a fresh model to each scene. The network has already learned, from many training scenes, how to turn images and their camera positions into a cloud of 3D Gaussians that can be rendered from new viewpoints.

Standard 3D Gaussian splatting works differently. It starts from a sparse point cloud and then optimizes the position, shape, color and opacity of every Gaussian for that one scene, repeatedly rendering and comparing against the input photos. The pixelSplat authors point out that this original approach needs dozens of images per scene and relies on pruning and splitting steps that are not differentiable, so they cannot be dropped into a network that is trained end to end. Feed-forward methods replace that per-scene loop with a single prediction.

How pixelSplat predicts Gaussians from two photos

pixelSplat, one of the first feed-forward splatting models, takes a pair of images and outputs a set of 3D Gaussians in a single pass. Each pixel's feature vector predicts a Gaussian, and the scene is the union of Gaussians from both views.

The first hurdle is scale. Camera poses from structure-from-motion are only known up to an unknown scale for each scene, so a network looking at one image alone cannot guess depths that match those poses. pixelSplat solves this with an epipolar encoder: for each pixel it searches along the matching line in the other image, finds correspondences and attaches triangulated depth information to the feature, so the prediction carries the right scale. A self-attention step then spreads that depth to regions that are visible in only one image.

The second hurdle is training. Regressing Gaussian centers directly gets stuck, because, as the paper explains, gradients vanish once a Gaussian is more than a few standard deviations from where it should be. pixelSplat instead predicts a probability distribution over depth buckets, samples a depth, and sets the Gaussian's opacity to the probability of the bucket it came from. Correct depths become more opaque and get reinforced; wrong ones fade. In the paper's ablations, removing the epipolar encoder cut image quality sharply, and removing probabilistic sampling also hurt it.

Cost volumes: how MVSplat made it faster

MVSplat takes a different route to depth. It builds a cost volume by plane sweeping: it tests a series of candidate depth planes and measures how similar the features from different views look at each one. Where views agree, a surface is likely, so the cross-view similarity gives a direct geometric cue for where to place each Gaussian center. The other Gaussian parameters are learned jointly, using only photometric supervision from images.

The MVSplat authors report that this design uses ten times fewer parameters than pixelSplat, infers more than twice as fast, reaches 22 frames per second on the RealEstate10K and ACID benchmarks, and generalizes better across datasets. The broader lesson is that explicit multi-view geometry inside the network makes the learning problem easier.

Why more input views is the hard part

Early feed-forward models worked from two or a few views. Real captures often have dozens or hundreds of frames, and feed-forward models struggle there. The ZPressor paper describes the limit as encoder capacity: as views are added, these models either lose quality or run out of memory. Because many methods emit Gaussians per pixel, more views also means many redundant Gaussians.

ZPressor compresses the input by splitting views into a small set of anchor views and a larger set of support views, then using cross-attention to pour information from the support views into the anchors. Its authors report scaling existing feed-forward models to more than 100 input views at 480p on an 80 GB GPU.

A newer paper, CrossPressor, argues that fixed anchors create their own bottleneck: each anchor must absorb more and more support views as the sequence grows. Its training-free fix keeps every anchor but routes only selected spatial evidence from the support views, cycling coverage so every view is still represented. The authors report quality close to dense fusion, with cross-view fusion up to about 26 times faster and end-to-end runs up to about 1.4 times faster on long sequences.

When to use feed-forward splatting versus classic optimization

Feed-forward splatting is attractive when speed matters more than the last bit of fidelity: previewing a capture moments after it is taken, building scenes from very few photos, or processing many scenes in bulk. Classic per-scene optimization still makes sense when you have plenty of images and time and want the best possible result for one scene. Feed-forward output also inherits the training data's assumptions, so results depend on how close a new scene is to what the model learned from, and like any structure-from-motion pipeline it still works in arbitrary units unless real-world scale is added separately.