Learning Factorized Diffusion Policies for Conditional Action Diffusion

Video

Method Overview

In this work, we propose a novel theoretical framework Factorized Diffusion Policies (FDP) for learning action diffusion models that decouples observational modalities for prioritization. At its core, FDP learns a residual model using some input modalities that have been omitted while training a base model with prioritized inputs. The base and residual model outputs are then composed to obtain samples from the full conditional action distribution. In addition, we present an architecture that enables efficient learning of the residual model in the FDP framework. We demonstrate that prioritization of modalities may yield significant gains in sample efficiency and naturally improves policy robustness to distribution shifts in the residual observations.

Real Robot Setup

Tasks

We evaluate Factorized Diffusion Policy (FDP) and the Diffusion Policy (DP) baseline across four real-world domains and report their task success rates. The domains are: Close Drawer as a simple task where the robot has to push the drawer; Put Block in Bowl that assesses the policy’s ability to perform precise pick-and-place actions; Pour in Bowl to evaluate the policy’s dexterity in operating near joint limits and Fold Towel to assess effectiveness in manipulating deformable objects.

Data Collection

We collect 50 demonstrations per domain on a Franka FR3 robot using a 6D space mouse, recording both proprioceptive and visual observations from two cameras—one mounted on the gripper and a static camera covering the workspace.

Real Robot Results

The trained policies are evaluated on four task variations in each domain: default: an in-distribution setup matching the conditions used during demonstration collection; color: the object’s color is altered to test robustness to visual appearance changes; distractor: novel, unseen objects such as vegetation props and soft toys are added to the scene to introduce clutter; and occlusion: visual input is intermittently blocked during policy rollout to simulate partial observability.