Gen4P
Gen4P

Gen4P: Video Generators are 4D Perceivers

Yitong Xia1, Zeren Jiang2, Chuanxia Zheng1
1Physical Vision Group (PVG), Nanyang Technological University (NTU)
2Visual Geometry Group (VGG), Oxford University
Paper (TBD) arXiv (TBD) Code (TBD) BibTeX

Faithful structure preservation and motion tracking on challenging scenes like flamingo.

Method

TL;DR: Gen4P repurposes a video generator for feed-forward 4D perception, with single-pass encoding, on-demand decoding for spatial flexibility, continuous-time embeddings for temporal flexibility, and a frozen depth LoRA as a plug-in geometry prior.

Overview of the Gen4P pipeline
Overview of Gen4P. The input video is encoded into scene tokens F in a single pass by the Video Generative Encoder, which consists of a frozen VAE encoder and DiT, an optional frozen Depth-LoRA, and a trainable 4D-LoRA. The cross-attention Query Decoder is queried independently for each query q(u, v, tsrc, ttgt, tcam), where the query timestamps are parameterized as continuous time embeddings. The decoder outputs the corresponding 3D and 2D point attributes, enabling the full construction of 3D tracks, depth maps and camera poses.

Qualitative Results of 3D Tracking

Additional Qualitative Results

Confidence-based filtering are applied for the best visual quality.
We tune with our best efforts to maintain the visual quality on all methods.

Acknowledgment

This research is supported by the NTU SUG-NAP and National Research Foundation, Singapore, under its NRF Fellowship Award NRF-NRFF17-2025-0009. We would like to acknowledge the High Performance Computing Centre of NTU Singapore, for providing the computing resources, facilities, and services that have contributed significantly to this work. Zeren Jiang is supported by Clarendon Scholarship.

BibTeX

@misc{xia2026gen4p,
  title={Gen4P: Video Generators are 4D Perceivers},
  author={Yitong Xia and Zeren Jiang and Chuanxia Zheng},
  year={2026},
  eprint={2610.XXXXX},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.XXXXX}
}