D4RT: Teaching AI to see the world in four dimensions

D4RT (Dynamic 4D Reconstruction and Tracking) is a unified AI model that reconstructs and tracks dynamic 3D scenes across space and time from 2D video input.

• •D4RT uses a single encoder-decoder Transformer architecture, replacing the fragmented pipeline of separate models for depth, motion, and camera estimation
• •A flexible query mechanism answers one core question: where a given pixel is located in 3D space at an arbitrary time from a chosen camera viewpoint
• •Queries are processed in parallel on modern hardware, making D4RT 18x to 300x faster than previous state-of-the-art methods
• •A one-minute video is processed in roughly five seconds on a single TPU chip, compared to up to ten minutes for prior methods
• •D4RT supports point tracking, point cloud reconstruction, and camera pose estimation through a single unified interface