DriveX - Foundation Models for V2X-based Cooperative Autonomous Driving 🚗

Introduction

The DriveX Workshop explores the integration of foundation models and V2X-based cooperative systems to improve perception, planning, and decision-making in autonomous vehicles. While traditional single-vehicle systems have advanced tasks like 3D object detection, emerging challenges like holistic scene understanding and 3D occupancy prediction require more comprehensive solutions. Collaborative driving systems, utilizing V2X communication and roadside infrastructure, extend sensory range, provide hazard warnings, and improve decision-making through shared data. Simultaneously, foundation models like Vision-Language Models (VLMs) offer generalization abilities, enabling zero-shot learning, open-vocabulary recognition, and scene explanation for novel scenarios. Recent advancements in end-to-end systems and foundation models like DriveLLM further enhance autonomous systems. The workshop aims to bring together experts to explore these technologies, address challenges, and advance road safety.

Topics

Foundation Models for Cooperative Autonomous Driving and Intelligent Transportation Systems
Vision-Language Models (VLMs) for Traffic Scene Understanding
Large Language Model (LLM)-assisted Cooperative Systems
Communication-Efficient Cooperative Perception for Autonomous Vehicles
Efficient and Intelligent Vehicle-to-everything (V2X) Communication
Dataset Curation and Data Labeling for Autonomous Driving
Datasets and Benchmarks for Foundation Models and Cooperative Perception
3D Object Detection and Semantic Segmentation of Vulnerable Road Users (VRUs)
3D Occupancy Prediction and Scene Understanding
End-to-end Perception and Real-time Decision-Making Systems
Vehicle-to-Infrastructure (V2I) Interaction
Safety and Standards in Autonomous Driving

08:00 - 08:10	Opening Remarks & Welcome
08:10 - 08:30	Intro: Walter Zimmer (TUM) Roadside 3D Perception for Autonomous Driving
08:30 - 08:50	Keynote 1: (remote) Prof. Dr. Cyrill Stachniss (Uni. of Bonn) Semantic Perception for Autonomous Vehicles
08:50 - 09:10	Keynote 2: Prof. Dr. Manmohan Chandraker (Uni. of California San Diego) Foundational Models for V2X Intersection Safety
09:10 - 09:30	Keynote 3: Dr. Mingxing Tan (Waymo) VLMs for E2E Autonomous Driving
09:30 - 09:50	Keynote 4: Prof. Dr. Philipp Krähenbühl (Uni. of Texas at Austin) Robust Autonomy Emerges from Self-Play
09:50 - 10:00	Coffee Break (ExHall D) & Sponsorship Announcements (AbakaAI)
10:00 - 10:20	Keynote 5: (remote) Maria Lyssenko (TUM & BOSCH) Functional Evaluation of Perception Performance for Automated Driving (AD)
10:20 - 10:40	Keynote 6: Prof. Dr. Trevor Darrell (Uni. of California Berkeley) Autonomous Driving and Robotics
10:40 - 11:00	Keynote 7: Prof. Dr. Marco Pavone (NVIDIA & Stanford Uni.) Foundation Models for Autonomous Driving
11:00 - 11:20	Keynote 8: Prof. Dr. Laura Leal-Taixé (NVIDIA & TUM) Repurposing Generative Models for 3D Data
11:20 - 11:40	Keynote 9: (remote) Prof. Dr. Jiaqi Ma (Uni. of California Los Angeles) A Multi-Agent Future of Mobility with Intelligent Vehicles and Infrastructure
11:40 - 12:00	Q&A and Panel Discussion I
12:00 - 13:00	Lunch Break (ExHall C)
13:00 - 13:20	Keynote 10: Dr. Katie Luo (Stanford Uni.) Beyond Onboard Sensors: Leveraging Auxiliary Information for Self-Driving Perception
13:20 - 13:40	Keynote 11: Prof. Dr. Chen Feng (NYU) CoPerception and Retrospection for Autonomous Driving
13:40 - 14:00	Keynote 12: (remote) Prof. Dr. Wolfram Burgard (Uni. of Technology Nuremberg, UTN) TBD
14:00 - 14:20	Keynote 13: Dr. Gianluca Corrado (Wayve) Foundation Models for Embodied AI
14:20 - 14:40	Keynote 14: Prof. Dr. Angela Dai (TUM) From Quantity to Quality for 3D Perception
14:40 - 15:00	Keynote 15: Prof. Dr. Daniel Cremers (TUM) Self-supervised Learning for Dynamic Scene Understanding
15:00 - 15:40	Q&A and Panel Discussion II
15:40 - 15:50	Coffee Break (ExHall D)
15:50 - 16:00	Oral Paper Presentation 1: Johannes Spoecklberger Exploring Modality Guidance to Enhance VFM-based Feature Fusion for UDA in 3D Semantic Segmentation
16:00 - 16:10	Oral Paper Presentation 2: Hsu-kuang Chiu V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models
16:10 - 16:20	Oral Paper Presentation 3: Tonko E. W. Bossen Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehicles
16:20 - 16:30	Oral Paper Presentation 4: Esteban Rivera V3LMA: Visual 3D-enhanced Language Model for Autonomous Driving
16:30 - 16:40	Oral Paper Presentation 5: Zhihao Zhao CooPre: Cooperative Pretraining for V2X Cooperative Perception
16:40 - 16:50	Paper Award Ceremony: Prof. Dr.techn. Daniel Watzenig (Graz Uni. of Technology) Best Paper Awards (Announcement & Handover of Prizes) + Group Photo
16:50 - 17:10	Keynote 16: (remote) Prof. Dr. Manabu Tsukada (Uni. of Tokyo) Enabling Cooperative End-to-End Autonomous Driving: Feature Sharing, Adaptation, and Preference Learning
17:10 - 17:20	Competition Winner Presentation: Chuheng Wei (UCLA) HeCoFuse: Cross-Modal Complementary V2X Cooperative Perception with Heterogeneous Sensors
17:20 - 17:30	Competition Award Ceremony: Walter Zimmer (TUM) Competition Winner Awards (Announcement & Handover of Prizes) + Group Photo
17:30 - 17:30	Walter Zimmer (TUM) Announcement of Evening Social Event (Location & Time), Group Picture of all Organizers and Speakers
17:30 - 18:00	Poster Session (Room: ExHall D, Boards: #121-#127) & Networking
19:00 - 21:00	Workshop Banquet (Social Mixer, Networking, Dinner) - Free for Workshop Organizers, Speakers and Invited Guests (Location will be announced during the workshop)

Paper Track

We accept novel papers (8 pages excl. references) for publication in the proceedings, and shorter extended abstracts (4 pages excl. references). Full papers should use the official LaTeX or Typst CVPR 2025 template. For extended abstracts we recommend to use the same template. Submission Portal: OpenReview

~~Submission Opens~~	~~February 26, 2025~~
Paper Abstract Submission Deadline	~~March 25, 2025 (23:59 PST)~~ March 26, 2025 (23:59 GMT)
Paper Submission Deadline	~~March 25, 2025 (23:59 PST)~~ March 26, 2025 (23:59 GMT)
Notification to Authors	~~March 31, 2025~~
Camera-ready Submission	~~April 7, 2025 (23:59 GMT)~~

Paper Awards

🏆 Best Paper Award: prize money ($500 in cash) 💰 and a certificate of recognition 📜
🏅 Best Paper Runner-Up Award: prize money ($400 in cash) 💰 and a certificate of recognition 📜
🎖️ Best Application Paper Award: prize money ($300 in cash) 💰 and a certificate of recognition 📜

Accepted Papers

CooPre: Cooperative Pretraining for V2X Cooperative Perception

Seth Z. Zhao, Hao Xiang, Chenfeng Xu, Xin Xia, Bolei Zhou, Jiaqi Ma

PDF

V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models

Hsu-kuang Chiu, Ryo Hachiuma, Chien-Yi Wang, Stephen F Smith, Yu-Chiang Frank Wang, Min-Hung Chen

PDF

V3LMA: Visual 3D-enhanced Language Model for Autonomous Driving

Esteban Rivera, Jannik Lübberstedt, Nico Uhlemann, Markus Lienkamp

PDF

Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehicles

Tonko Emil Westerhof Bossen, Andreas Møgelmose, Ross Greer

PDF

Exploring Modality Guidance to Enhance VFM-based Feature Fusion for UDA in 3D Semantic Segmentation

Johannes Spoecklberger, Pedro Hermosilla, Wei Lin, Sivan Doveh, Horst Possegger, Muhammad Jehanzeb Mirza

PDF

VPOcc: Exploiting Vanishing Point for 3D Semantic Occupancy Prediction

Junsu Kim, Junhee Lee, Ukcheol Shin, Jean Oh, Kyungdon Joo

PDF

Investigating Vision-Language Model for Point Cloud-based Vehicle Classification

Yiqiao Li, Jie Wei, Camille Kamga

PDF

Challenge

We host a 10-week-long challenge based on the TUMTraf V2X Cooperative Perception Dataset (CVPR'24) which includes high-quality, real-world V2X perception data for the cooperative 3D object detection and tracking task in autonomous driving. The dataset is available here. We provide a dataset development kit to work with the dataset. Our TUMTraf V2X dataset contains:

Data collected by 9 sensors simultaneously from onboard and roadside sensors.
2,000 labeled point clouds and 5,000 labeled images.
30k 3D bounding boxes with track IDs.
Challenging scenarios: near-miss and traffic violation events, overtaking and U-turn maneuvers.
HD maps of the driving domains.
Labels in OpenLABEL standard.
A dev kit to load, preprocess, visualize, convert labels, and to evaluate perception methods.

Competition Timeline:

~~Dataset Final Release:~~	~~December 31, 2024~~
~~Competition Announcement~~	~~January 8, 2025~~
~~Competition Starts (Leaderboard opens on EvalAI)~~	~~March 17, 2025~~
Competition Ends (Leaderboard closes on EvalAI)	~~May 26, 2025 (23:59 GMT)~~
~~Abstract Submission Deadline (via email)~~	~~May 26, 2025 (23:59 GMT)~~
~~Challenge Report (4-8 pages) Submission Deadline (via email)~~	~~June 1, 2025 (23:59 GMT)~~
Notification to Participants (1st, 2nd, 3rd Prize Winner Announcement)	June 6, 2025
Video Presentation (10 min) Submission (via email)	June 11, 2025 (23:59 GMT)

Evaluation Metrics:

Precision: Ratio of correctly detected objects to the total number of detected objects, measuring accuracy of positive predictions.
Recall: Ratio of correct detections to the total number of actual objects, assessing the model's ability to find all relevant instances.
3D IoU: Ratio of intersection volume to the union volume between predicted and ground truth 3D box, measuring spatial overlap.
Position RMSE: Square root of mean squared differences between predicted and GT positions, evaluating localization accuracy.
Rotation RMSE: Square root of mean squared differences between predicted and GT rotations, quantifying angular estimation errors.
3D mAP: Average precision across intersection-over-union (IoU) thresholds in 3D detection, reflecting overall detection performance.

Challenge Submission:
CoopDet3D generates detection (inference) files in OpenLABEL (.json) format (see example). In case you are developing your own model, please output the detection results in the OpenLABEL format following this schema. After you have generated the detection files for the test set (100 frames), please combine them into a single OpenLABEL (.json) file using this script. Test the evaluation locally on your machine using our evaluation script in the dev-kit, before submitting to the leaderboard. Finally, submit the accumulated OpenLABEL prediction file (submission.json) to the EvalAI Leaderboard platform. The best-performing teams will be invited to present their solutions during the workshop, and the winners will receive prizes and recognition for their contributions to the field.

Challenge Awards

🏆 Challenge Winner Award (1st Place): Gold medal & prize money ($500 in cash) 💰
🥈 Challenge Runner-up Award (2nd Place): Silver medal & prize money ($400 in cash) 💰
🥉 Challenge Third Place Award (3rd Place): Bronze medal & prize money ($300 in cash) 💰