Multimodal Perception and Comprehension of Corner Cases in Autonomous Driving

Introduction

This workshop aims to bridge the gap between state-of-the-art autonomous driving techniques and fully intelligent, reliable self-driving agents, particularly when confronted with corner cases, rare but critical situations that challenge limits of reliable autonomous driving. The advent of Multimodal Large Language Models (MLLMs), represented by GPT-4V, demonstrates the unprecedented capabilities in multimodal perception and comprehension even under dynamic street scenes. However, leveraging MLLMs to tackle the nuanced challenges of self-driving still remains an open field. This workshop seeks to foster innovative research in multimodal perception and comprehension, end-to-end driving systems, and the application of advanced AIGC techniques to autonomous systems. We conduct a challenge comprising two tracks: corner case scene understanding and corner case scene generation. The dual-track challenge is designed to advance reliability and interpretability of autonomous systems in both typical and extreme corner cases.

Challenges

Track 1: Corner Case Scene Understanding

This track is designed to enhance perception and comprehension abilities of MLLMs for autonomous driving, focusing on global scene understanding, regional reasoning, and actionable suggestions. With the CODA-LM dataset consisting of 5,000 images with textual descriptions covering global driving scenarios, detailed analyses of corner cases, and future driving recommendation, this track seeks to promote the development of more reliable and interpretable autonomous driving agents.

• Datasets: Please refer to CODA-LM for detailed dataset introduction and dataset downloads.
• Please find more information on the Track1 page.

Track 2: Corner Case Scene Generation

This track focuses on improving the capabilities of diffusion models to create multi-view street scene videos that are consistent with 3D geometric scene descriptors, including the Bird's Eye View (BEV) maps and 3D LiDAR bounding boxes. Building on MagicDrive for controllable 3D video generation, this track aims to advance scene generation for autonomous driving, ensuring better consistency, higher resolution, and longer duration.

• Datasets: Please refer to MagicDrive for detailed dataset introduction and dataset downloads (i.e., nuScenes dataset).
• Please find more information on the Track2 page.

Call for Papers

Overview. This workshop aims to foster innovative research and development in multimodal perception and comprehension of corner cases for autonomous driving, critical for advancing the next-generation industry-level self-driving solutions. Our focus encompasses a broad range of cutting-edge topics, including but not limited to:

• Corner case mining and generation for autonomous driving.
• 3D object detection and scene understanding.
• Semantic occupancy prediction.
• Weakly supervised learning for 3D Lidar and 2D images.
• One/few/zero-shot learning for autonomous perception.
• End-to-end autonomous driving systems with Large Multimodal Models.
• Large Language Models techniques adaptable for self-driving systems.
• Safety/explainability/robustness for end-to-end autonomous driving.
• Domain adaptation and generalization for end-to-end autonomous driving.

Submission tracks. All submissions should be anonymous, and reviewing is double-blind. We encourage two types of submissions:

• Full workshop papers not previously published or accepted for publication in the substantially similar form in any peer-reviewed venues including journals, conferences or workshops. Papers are limited to 14 pages, including both figures and tables, in ECCV format with extra pages containing only cited references allowed. Accepted papers will be part of the official ECCV proceedings.
  [Download LaTex Template]
  [OpenReview Submission Site]
• Extended abstracts not previously published or accepted for publication in substantially similar form in any other peer-reviewed venues including journals, conferences or workshops. Papers are limited to 4 pages, and will NOT be included in official ECCV proceedings (non-archival), which are allowed for re-submission to later conferences.
  [Download LaTex Template]
  [OpenReview Submission Site]

Accepted papers (full track).

• Paper ID 2: AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving - Daniel Bogdoll, Iramm Hamdard, Lukas Roessler, Felix Geisler, Muhammed Bayram, Felix Wang, Jan Imhof, Miguel de Campos, Anushervon Tabarov, Yitian Yang, Martin Gontscharow, Hanno Gottschalk, J. Marius Zoellner
• Paper ID 3: On Camera and LiDAR Positions in End-to-End Autonomous Driving - Malte Stelzer, Jan Pirklbauer, Jan Bickerdt, Volker Patricio Schomerus, Jan Piewek, Thorsten Bagdonat, Tim Fingscheidt
• Paper ID 4: ProGBA: Prompt Guided Bayesian Augmentation for Zero-shot Domain Adaptation - Jian Zou, Guanglei Yang, Tao Luo, Chun-Mei Feng, Wangmeng Zuo
• Paper ID 7: ReGentS: Real-World Safety-Critical Driving Scenario Generation Made Stable - Yuan Yin, Pegah KHAYATAN, Eloi Zablocki, Alexandre Boulch, Matthieu Cord
• Paper ID 8: Loop Mining Large-Scale Unlabeled Data for Corner Case Detection in Autonomous Driving - Jiawei Zhao, Yiting Duan, Jinming Su, yangwangwang, Tingyi Guo, Xingyue Chen, Junfeng Luo
• Paper ID 9: HumanSim: Human-Like Multi-Agent Novel Driving Simulation for Corner Case Generation - Lingfeng Zhou, Mohan Jiang, Dequan Wang
• Paper ID 11: Talk to Parallel LiDARs: A Human-LiDAR Interaction Method Based on 3D Visual Grounding - Yuhang Liu, Boyi Sun, Yishuo Wang, Jing Yang, Xingxia Wang, Fei-Yue Wang
• Paper ID 16: RoSA Dataset: Road construct zone Segmentation for Autonomous Driving - JINWOO KIM, Kyounghwan An, Donghwan Lee
• Paper ID 17: A Multimodal Hybrid Late-Cascade Fusion Network for Enhanced 3D Object Detection - Carlo Sgaravatti, Roberto Basla, Riccardo Pieroni, Matteo Corno, Sergio M. Savaresi, Luca Magri, Giacomo Boracchi

Accepted papers (abstract track).

• Paper ID 2: Challenge report: Track 2 of Multimodal Perception and Comprehension of Corner Cases in Autonomous Driving - Zhiying Du, Zhen Xing
• Paper ID 3: DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes - Jianbiao Mei, Yukai Ma, Xuemeng Yang, Licheng Wen, Tiantian Wei, Min Dou, Botian Shi, Yong Liu
• Paper ID 4: Two-Stage LVLM system: 1st Place Solution for ECCV 2024 Corner Case Scene Understanding Challenge - Ying Xue, Haiming Zhang, Yiyao Zhu, Wending Zhou, Shuguang Cui, Zhen Li
• Paper ID 5: NexusAD: Exploring the Nexus for Multimodal Perception and Comprehension of Corner Cases in Autonomous Driving - Mengjingcheng Mo, Jingxin Wang, Like Wang, Haosheng Chen, Changjun Gu, Jiaxu Leng, Xinbo Gao
• Paper ID 6: DiVE: DiT-based Video Generation with Enhanced Control - Junpeng Jiang, Gangyi Hong, Lijun Zhou, Enhui Ma, Hengtong Hu, xia zhou, Jie Xiang, Fan Liu, Kaicheng Yu, Haiyang Sun, Kun Zhan, Peng Jia, Miao Zhang
• Paper ID 7: From Regional to General: A Vision-Language Model-Based Framework for Corner Cases Comprehension in Autonomous Driving - XU HAN, Yehua Huang, SongTang, Xiaowen Chu
• Paper ID 8: A Lightweight Vision-Language Model Pipeline for Corner-Case Scene Understanding in Autonomous Driving - Ying Cheng, Min-Hung Chen, Shang-Hong Lai
• Paper ID 9: Iterative Finetuning VLM with Retrieval-augmented Synthetic Datasets Technical Reports for W-CODA Challenge Track-1 from Team OpenDriver - Zihao Wang, Xueyi Li
• Paper ID 11: FORTRESS: Feature Optimization and Robustness Techniques for 3D Object Detection Systems - Caixin Kang, Xinning Zhou, Chengyang Ying, Wentao Shang, Xingxing Wei, Yinpeng Dong, Hang Su
• Paper ID 12: Adversarial Policy Generation in Automated Parking - Alessandro Pighetti, Francesco Bellotti, Riccardo Berta, Andrea Cavallaro, Luca Lazzaroni, Changjae Oh

Important Dates (AoE Time, UTC-12)

Schedule (Milano Time, UTC+2)

Invited Speakers

Organizers

Challenge Committee

Contact

  To contact the organizers please use w-coda2024@googlegroups.com (p.s. we recently fixed the delivery failure problem ^v^).

  Join our WeChat group for more information.