DeFM-WMPFrom terrain perception to quadruped locomotion.

From visible geometry to sustained locomotion

This work integrates pretrained DeFM depth representations into the recurrent world model of WMP and expands the training terrain curriculum. A learned readout combines spatial depth features with proprioceptive observations for world-model state estimation and control. Narrow bridges, stepping stones, planters and varied hurdles add geometric challenges to the shared training setup. This preview presents simulation progress on that integration and terrain expansion.

This article presents simulation rollouts alongside the complete path from depth observations to joint torques. The current DeFM checkpoint contributes successful level-8 clips on 12 terrain types. Four additional clips—double-row stepping stones and three fence courses—come from earlier WMP experiments. Together they form a 16-terrain project showcase, not a success-rate evaluation of one checkpoint.

01 / Watch the terrain traversals

Sixteen continuous simulation clips span stepping stones, bridges, planters, climbing, and fences. Each runs for 12 seconds at level 8 with a forward command, zero resets, and at least six metres of forward displacement. The clips document individual traversals, not success rates over repeated seeds.

02 / From depth observations to joint control

The system couples recurrent state estimation with a history-conditioned controller. The world model maintains memory from depth and proprioception; the actor combines that memory with recent observations and velocity commands. RSSM is an intermediate representation, followed by feature encoding, an action policy, and low-level control.

A spatial readout for control

The DeFM visual front end encodes depth observations as spatial tokens. A learned readout combines global and local features, then joins them with encoded proprioception to form the world-model observation representation. This connects terrain geometry with the robot’s posture and joint state, providing a shared basis for recurrent state estimation.

Action-conditioned prediction, observation-conditioned inference

At world-model update k, the RSSM first advances deterministic memory using the previous latent state and the intervening action sequence A. The current observation embedding then conditions the stochastic posterior. Current visual evidence updates z before influencing h through a subsequent recurrent transition. Between depth updates, the controller reuses the latest deterministic state.

During training, prior–posterior KL regularization accompanies depth reconstruction, proprioceptive reconstruction, and reward prediction. These heads train the representation; action selection does not require decoding a depth image.

Beyond RSSM: the controller and the robot

The actor concatenates an encoding of deterministic state h, an encoding of the five-frame observation history H, and the velocity command c. The stochastic state z is not a direct actor input. Twelve action outputs are scaled and offset by a reference pose to produce joint position targets; a PD controller converts them to torques. A separate privileged critic and an AMP discriminator support policy training through value estimation and motion-style rewards.

Visual memory, recent motion feedback, and velocity commands meet in the actor before becoming joint targets. Terrain information therefore reaches motion through a complete chain of state estimation, action selection, and low-level control.

03 / Beyond continuous ground

Narrow beams and stepping stones constrain support. Planters and ramps introduce abrupt changes in height. Fences and low passages demand coordination between leg clearance and body posture. Our training terrain combines these challenges within a shared curriculum, exposing the policy to the transitions between approach, traversal, and continued motion. Increasing difficulty extends learning from basic locomotion to more demanding terrain interactions.