Robots Do Not Need to Predict Every Pixel of Touch—They Need to Predict Contact Change

A robot can see a cup and still fail to feel that it is slipping.

That gap matters because contact-rich work is not solved by recognising an object alone. Inserting a plug, unscrewing a lid or pulling a key from a lock depends on tiny changes in pressure, rotation and alignment. Cameras may show almost the same scene while the physical interaction has already moved towards success or failure.

A robotics preprint submitted to arXiv on 30 September 2026 proposes an interesting change of target. Instead of asking a model to reconstruct the exact future image produced by a tactile sensor, TacDyn-WAM tries to predict the underlying pattern of contact change. The distinction sounds technical, but it points to a larger question for embodied AI: should a useful world model reproduce what a sensor will look like, or preserve the physical information needed to choose the next action?

Why touch is not just another camera

Many modern tactile sensors are camera-based. A soft surface presses against an object, and an internal camera records the resulting deformation. The output is an image, but its meaning is unlike an ordinary photograph. Small shifts in the contact point or force can produce large pixel changes. Conversely, an important physical event such as the start of a slip may occupy only a small region.

This creates a distribution-shift problem. Demonstrations usually show a robot making reasonably good contact. Once the learned policy makes a slight error, the tactile image may move outside that familiar range even though the broad evolution of the interaction is still predictable: an imprint is deepening, sliding, rotating or disappearing.

Reconstructing every future tactile pixel can therefore force the model to spend capacity on sensor appearance rather than the part of the signal that matters for control. It may also require iterative image-generation steps at inference time. The researchers behind TacDyn-WAM argue that a representation trained specifically to capture contact dynamics could be both more robust and faster.

What the model predicts instead

TacDyn-WAM separates vision and touch rather than squeezing both into one visual target space. Its visual expert predicts future visual features. A tactile expert predicts future tactile representations and their change from the current state at three time horizons. A separate read-only memory supplies information about the contact happening now.

The tactile representation, called TacRep, is learned from short tactile video clips. Parts of those clips are masked during training, so the encoder has to recover information about how the contact pattern moves over time. A second training signal preserves local spatial relationships without requiring exact pixel matching. In plain language, the model is encouraged to recognise the shape and direction of a contact event without memorising the precise look of the sensor surface.

The model predicts both the future representation and the difference from the present. That dual target is sensible. A final state on its own can hide whether the contact squeezed, slipped and returned to a similar position. A change vector without its starting point can be ambiguous. Keeping both offers a richer account of the interaction.

What the experiments reported

The authors evaluated the system in simulation and on a real robot. On the eight-task UniVTAC benchmark, which includes activities such as inserting a tube, pulling out a key and inserting an HDMI connector, TacDyn-WAM recorded an average success rate of 81.5% using the benchmark’s task demonstrations. The paper reports 100 roll-outs per task.

The result was not the highest number in the table. Two heavily pretrained comparison systems averaged 83.1% and 84.5%. That context matters. The authors’ claim is not that their method won every task, but that it approached those results without the same large-scale visuo-tactile pretraining.

The ablations are more informative than the headline score. Replacing TacRep with a reconstruction-oriented visual autoencoder reduced the average from 81.5% to 62.0%. Using static DINOv2 features produced 74.0%. Removing the tactile prediction pathway or the current-state memory also reduced performance. Within this experimental setup, that supports the design choice to preserve temporal contact change and current contact information separately.

For five physical-robot tasks, the researchers collected 60 teleoperated demonstrations per task and evaluated each policy over 20 trials per task. TacDyn-WAM averaged 71.0%; a version pretrained on about 6,000 selected real visuo-tactile trajectories averaged 85.0%. The tasks included stacking cups, removing and inserting a plug, unscrewing a cup lid and wiping a whiteboard.

The paper also reports an inference latency of 9.7 milliseconds per action on one A100 GPU under its test conditions. That figure is not a general speed guarantee, but it illustrates the computational case for predicting compact features in one forward pass instead of repeatedly generating future frames.

The useful lesson is about choosing the target

AI systems learn what their objectives reward. If a robot is rewarded for reconstructing tactile pixels, it may become excellent at reproducing sensor appearance without learning the most useful physical abstraction. If it is trained only on a single final outcome, it may miss the path taken to reach it. If touch is merged too early with vision, the stronger visual signal may dominate.

TacDyn-WAM’s deeper contribution is therefore not merely “add touch”. It is to ask which properties of touch deserve prediction:

• Direction: is the contact rotating, sliding, compressing or releasing?
• Timing: does that change occur immediately or over a longer horizon?
• Spatial structure: which part of the contact patch is changing?
• Current state: what is the robot feeling before it forecasts what comes next?

This is relevant beyond robotics. Whenever an AI system observes a high-dimensional signal, a photorealistic prediction is not automatically the best decision aid. A representation should discard nuisance variation while retaining changes that affect action. The difficult part is proving that the chosen abstraction has not discarded a rare but safety-critical signal.

Where the evidence stops

This is a first-version preprint, not a peer-reviewed conclusion about tactile robotics. The experiments were designed and reported by the same research team that proposed the method. Independent replication would strengthen the result.

The physical evaluation is also small: five tasks, 20 trials per task and one robot setup. Twenty trials can distinguish large performance gaps, but it is not enough to characterise rare failures confidently. A system that succeeds 17 times out of 20 may still hide brittle behaviours that appear under different objects, lighting, wear, contamination or operator demonstrations.

The authors name another important limitation: TacRep and the tactile expert were trained for one type of vision-based tactile sensor. The paper does not establish transfer to force sensors, taxel arrays or other tactile hardware. Nor does it show general-purpose dexterity across unbounded household or industrial environments.

There is also a deployment question. The reported latency was measured on an A100 GPU, not necessarily on the compute budget of a factory cell or mobile robot. A compact predictive target may reduce inference cost, but the complete system still has roughly 3.6 billion parameters according to the paper.

A practical reading checklist

When evaluating a claim that an embodied AI system “understands touch”, ask five questions:

1. What is the prediction target? Pixels, forces, features and contact-change representations encode different assumptions.
2. How far did execution drift from demonstrations? Robustness matters most after the first small error.
3. Were the strongest comparisons trained on comparable data? Pretraining scale can easily dominate an architectural difference.
4. How many independent physical trials were run? Simulation roll-outs and real-world attempts answer different questions.
5. What changed across sensors, objects and tasks? A method tied to one sensor or gripper is not yet a general tactile intelligence.

Touch makes prediction physical

Vision can tell a robot what an object appears to be. Touch can tell it whether the interaction is becoming stable, misaligned or unsafe. The research case made by TacDyn-WAM is that forecasting this evolution may be more useful than painting a perfect picture of the sensor’s future.

That is a promising design principle, not a finished verdict. The next tests should involve different tactile hardware, more varied physical environments, larger trial counts and independent replication. If the idea survives those tests, embodied AI may advance not by predicting every pixel it can observe, but by learning which physical changes matter before an action goes wrong.

---

Primary source: Enyi Wang and colleagues, “TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model”, arXiv preprint, version 1 submitted 30 September 2026. Figures and performance values above are author-reported unless otherwise stated.