From Sensing to Understanding: World Models for Semantic-Aware Collaborative Perception

Autonomous mobility systems increasingly rely on collaborative perception to overcome occlusion, limited field of view, and social navigation challenges in dynamic environments. However, effective collaboration is not simply about sharing more sensing data; it requires identifying information that is semantically valuable for a mobility agent’s task, decision-making, and evolving situational awareness. This talk explores how collaborative perception can move from extensive sensing to comprehensive understanding through world models. We begin with recent advances in vision-language models for semantic-aware perception, while highlighting key limitations: insufficient sensing data for reliable reasoning and the time-varying nature of perception evidence. To address these challenges, we introduce world models for evaluating collaboration policies that maintain reliable situational awareness as sensing coverage, mobility patterns, and communication conditions evolve. By predicting whether and how collaboration can improve semantic confidence under evolving sensing, mobility, and communication conditions, this approach transforms collaboration from reactive raw-data sharing into predictive, semantic-aware communication and policy reasoning, enabling autonomous systems to proactively identify efficient collaboration <a href="http://patterns.

Speaker(s):” target=”_blank” title=”patterns.

Speaker(s):”>patterns.

Speaker(s): Mushu Li

Room: 4152, Bldg: Centre for Environmental & Information Technology (EIT), University of Waterloo, Waterloo, Ontario, Canada