We study how LLMs and VLMs can work with spatial understanding, world models, and embodied control in a closed-loop agent system. Our goal is to connect reasoning with the physical world through observation, prediction, action, and feedback, with reliability and efficiency throughout.
We reconstruct scenes from images and video, including their geometry and motion. Our work spans semantic occupancy, Gaussian representations, streaming reconstruction, and implicit surfaces. A recurring question is what to remember as new observations arrive.
Video could not load. The original demo is available through the project link below.
Video could not load. The original demo is available through the project link below.
Our view is not a simple progression in which one representation replaces another. Depth, occupancy, points, Gaussians, and implicit fields offer different trade-offs. We want to retain the geometry, motion, and uncertainty that an agent needs to predict and act.
Open question Which information must persist across observations, and which detail can be discarded without compromising a later decision?
Describe complete space, then reduce the cost of supervision.
Introduced a tri-perspective representation for vision-based semantic occupancy, extending scene understanding beyond a single bird's-eye-view plane.
Combined multi-camera occupancy prediction with a pipeline that fuses LiDAR scans and reconstructs surfaces to generate dense occupancy labels, supporting richer scene understanding beyond object boxes.
Used signed distance fields and temporal image rendering to learn scene geometry from video supervision, reducing dependence on annotated 3D occupancy.
Used cylindrical tri-perspective features to recover semantic occupancy from LiDAR points, balancing 3D coverage with computational cost.
Allocate sparse scene elements where geometry and motion need them.
Represented scenes with sparse semantic Gaussians instead of dense grids, adapting spatial coverage to object scale while predicting 3D occupancy.
Used 3D Gaussians and a spatial-temporal field to reconstruct driving scenes and separate static and dynamic content through 4D consistency, without tracked 3D vehicle boxes.
Updated a Gaussian scene through ego motion, object motion, and newly observed regions. Forecasting is conditioned on current images to support continuous occupancy perception.
Introduced probabilistic Gaussian superposition and distribution-based initialization to reduce redundant overlap and representation of empty space.
Maintain geometry over time, from points to surfaces and motion.
Used causal attention and cached memory to turn offline visual geometry estimation into incremental, low-latency reconstruction.
Learned a queryable implicit scene field, connecting coherent surface reconstruction and novel-view rendering within one representation.
Represented dynamic scenes with shared motion bases, moving beyond independent point displacements toward structured 4D reconstruction.
Maintained explicit spatial pointers that aggregate nearby observations in 3D, giving streaming reconstruction a memory tied to scene structure.
Reconstructed metric-scale driving scenes from unposed camera sequences, supporting diverse camera configurations without external scale alignment.
GitHub snapshot:
·
Repository totals, not historical counts at each milestone.
Semantic Scholar citations:
·
Paper-level counts; coverage differs from Google Scholar. Unavailable is not zero.
We study how to represent and generate a changing world. SVG develops semantic representations for generation; OccWorld and OccSora model 3D scene dynamics; Astra predicts visual sequences under different actions.
Video could not load. The original demo is available through the project link below.
Video could not load. The original demo is available through the project link below.
Video could not load. The original demo is available through the project link below.
SVG studies semantic representations for generation; OccWorld and OccSora model 3D scene dynamics; Astra explores action-conditioned video. These are complementary strands, not a single chain of inherited models. Our hypothesis is that useful world models need both a strong generative foundation and predictions that distinguish the consequences of actions.
Open question Can the model expose when a prediction is unreliable, and can an agent use that signal to choose a better action?
Move from a static 3D state to possible scene evolutions.
Extended occupancy from a description of the present to a generative world model that jointly forecasts future scenes and ego motion.
Combined a 4D occupancy tokenizer with diffusion to generate temporally coherent scene sequences conditioned on a requested trajectory.
Give visual generation a semantically structured latent space.
Combined frozen visual features with a residual detail branch for latent diffusion without a VAE. SVG develops the generative foundation; action-conditioned dynamics is a distinct research problem.
Ask how an observation changes when the action changes.
Developed action-conditioned, streaming video prediction across driving, manipulation, and camera control, broadening interactive world modeling beyond one scenario.
GitHub snapshot:
·
Repository totals, not historical counts at each milestone.
Semantic Scholar citations:
·
Paper-level counts; coverage differs from Google Scholar. Unavailable is not zero.
In autonomous driving, we connect scene geometry and future prediction to trajectory planning. Recent work also studies language instructions and online reconstruction. We are interested in how an agent revises its plan when new observations disagree with its predictions.
Video could not load. The original demo is available through the project link below.
GenAD explores joint future generation; GaussianAD and DVGT-2 ground plans in 3D structure; Vega adds language instructions. Doe-1 explores a loop between driving and world modeling. Our next challenge is reliable recovery when observations contradict a plan: offline scores or a model-based loop alone do not establish real-world reliability.
Open question When reality departs from the prediction, how should the agent detect the mismatch, update its understanding, and replan?
Bring prediction and planning into the same generative problem.
Cast driving as future generation, jointly modeling other agents' motion and ego trajectories in a structured latent space.
Connect spatial representations and imagined futures to decisions.
Unified observation, description, and action tokens for visual question answering, future generation, and motion planning, exploring a closed loop between a driving model and a world model.
Connected semantic Gaussians, future scene forecasting, and ego trajectories in an end-to-end driving model, retaining a sparse spatial representation through decision-making.
Ground requested behavior in instructions and a reconstructed scene.
Connected instructions, action plans, and generated future observations so the same scene can support different requested behaviors.
Extended driving geometry to joint online reconstruction and trajectory planning, with a sliding-window cache for continuous operation.
GitHub snapshot:
·
Repository totals, not historical counts at each milestone.
Semantic Scholar citations:
·
Paper-level counts; coverage differs from Google Scholar. Unavailable is not zero.
Across understanding, prediction, and action, we need models that can reason over rich observations, and ways to examine when their outputs should be trusted.
Our aim is to give agents a cognitive core that connects perception, language, reasoning, and decisions. Our current work develops two ingredients: efficient multimodal inference and spatially grounded understanding.
Our current entry point is generated-content verification: detecting synthetic images and identifying visible failures in generated videos, with evidence people can inspect.
Looking ahead, we want to extend verification to world-model predictions and agent action risks. These remain open goals, not safety guarantees established by the current work.
Shape the embedding space with more informative training examples.
Earlier work on metric learning generated label-preserving examples with adaptive difficulty, exploring how training samples shape the geometry of a learned representation.
Retain the visual evidence that matters to the question.
Used text-guided visual token pruning and recycling without extra training to reduce multimodal inference cost while retaining information relevant to the question.
Give models spatial evidence to reason with, and make generated-content failures inspectable.
Detected human-perceivable artifacts in generated videos, grounding explanations in spatial regions and timestamps. This provides evidence for content verification, not a guarantee of physical-agent safety.
Unified image generation and generated-image detection, using interpretable detector feedback to improve generation while strengthening detection. The focus is generated-content analysis, not general agent alignment.
Combined semantic clustering and geometric features into compact 3D proxies, then aligned them with a vision-language model for spatial question answering and grounding. This develops a spatial reasoning ingredient for agents.
GitHub snapshot:
·
Repository totals, not historical counts at each milestone.
Semantic Scholar citations:
·
Paper-level counts; coverage differs from Google Scholar. Unavailable is not zero.
Our longer-term aim is a loop in which models inform action, actions reveal new evidence, and that evidence improves both the model and the agent.
Bring spatial memory and dynamic prediction together, so an agent can maintain an evolving model of its environment.
Use predicted outcomes to guide decisions, then learn from the gap between imagined and observed consequences.
Develop compact representations and efficient inference, alongside ways to inspect predictions and identify their limitations.
I am a postdoctoral fellow in EECS at UC Berkeley, affiliated with BAIR and Berkeley DeepDrive, working with Kurt Keutzer. I received my Ph.D. in Automation from Tsinghua University, advised by Jie Zhou and Jiwen Lu, and my B.S. in Physics from Tsinghua.
I welcome conversations with students and collaborators interested in world models, embodied agents, and reliable physical intelligence. For in-person or remote research internships at BAIR, please get in touch.
wzzheng@berkeley.edu
1,363
576