Introducing OctoSense: A Self-Supervised Learning Framework for Multimodal Robot Perception
Researchers have unveiled OctoSense, an open-source sensor platform designed for advanced multimodal robot perception. This platform integrates a variety of sensors, including stereo RGB and event cameras, LiDAR, a thermal camera, and inertial measurement units, complemented by RTK-corrected GPS and proprioception data from robotic systems. The OctoSense dataset comprises 59 hours of time-synchronized driving data captured in diverse environments and under varying conditions, including low-quality sensor scenarios. The study introduces a 'late-fusion' masked autoencoder that employs modality-specific tokenizers to effectively process and learn from the heterogeneous data, achieving rapid inference and outperforming traditional image-only models in tasks such as optical flow estimation, depth estimation, semantic segmentation, and ego-motion prediction. The architecture demonstrates resilience in challenging conditions, including nighttime and degraded sensory inputs. The dataset, code, and supplementary materials are available on the project's website.
arXiv AIAnthony Bisulco, Jeremy Wang, Kostas Daniilidis et al.Read → 