DexForesight
Grounding Dexterous Policies with
Implicit Spatial Foresight
* Corresponding author
Abstract
Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand.
Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy.
During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation.
At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline.
Real-robot experiments
Three dexterous manipulation tasks on the Sharpa platform, requiring precise finger coordination, object alignment, and controlled contact.
Close Box
Coordinate both hands to align the lid with the box and press it closed.
Insert Battery
Align a battery with its holder and guide it into place through precise finger positioning.
Screw On Lid
Stabilize the cup with one hand while the other aligns and rotates the lid to tighten it.
The DexForesight model
Learn how the world changes.
Transfer that knowledge into action.
DexForesight separates learning predictive spatial representations from generating actions. A spatial world model learns hand and scene dynamics; a policy absorbs those representations through query alignment.
Learn action-conditioned spatial dynamics
The world model combines RGB, offline depth, language, robot state, and kinematic hand geometry. Its Hand Flow Expert predicts 3D hand-point motion, while its Visual Dynamics Expert predicts changes in image features. Hand geometry provides supervision even for fingers hidden from the cameras.
Align queries. Retain the policy.
After world-model training, its parameters are frozen. Policy queries are aligned with the predictive world queries through token and relational objectives, then supplied to the action expert. At inference, the world model and future decoders are removed.
Data & training
Approximately 370 hours of demonstrations across 60 tasks cover spatial arrangement, assembly and contact, rotation and articulation, deformable objects, and tool use. A 100-hour subset adds scene depth estimated offline from stereo RGB sequences.
Warmup
Adapt the backbone and prediction modules to hand-motion and visual objectives.
Dynamics pretraining
Learn action-conditioned hand–scene dynamics from demonstration trajectories.
Depth injection
Refine spatial representations with offline depth and trainable fusion modules.
Evaluation results
Evaluated on 11 DexJoCo simulation tasks and three real-robot dexterous manipulation tasks: Screw On Lid, Close Box, and Insert Battery.
Simulation success
+11.1 points over the base policyReal-robot success
+8.3 points over the strongest baselineInference overhead
230.4 ms vs. 227.4 ms for QwenVLARetaining spatial foresight
Does the policy retain predictive knowledge after the world model is removed? A linear probe of frozen policy features measures future 3D hand-point displacement error. DexForesight yields lower error than the architecture-matched QwenVLA action-only policy at every evaluated horizon.
Future hand-motion error
0.54 s horizon · lower is better26.3% lower displacement error at the longest evaluated horizon. Features are extracted from the policy alone.
In simulation, query alignment also outperforms directly attaching predictive heads to the base policy: 60.5% overall success versus 53.1% for the strongest direct-prediction control.
Conclusion
Spatial foresight for precise, coordinated dexterous manipulation.
DexForesight grounds policy latents in predictive hand–scene dynamics learned from offline depth, kinematic hand geometry, and demonstrated controls. Query alignment transfers these dynamics into a policy that uses only RGB, language, and robot state at deployment.
Read the full paperBibTeX
@misc{chen2026dexforesight,
title = {{DexForesight}: Grounding Dexterous Policies
with Implicit Spatial Foresight},
author = {Chen, Zhongxi and Yang, Yi and Chen, Xinyi and
Ye, Mingzhe and Li, Yaping and Liu, Zhihong and
Xu, Congsheng and Wang, Haonan and Deng, Zhijie},
year = {2026},
eprint = {2611.11166},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2611.11166}
}