MMBind Framework Achieves Breakthroughs in Multimodal Learning from Incomplete Data

The mDOT Center’s TR&D3 team has developed MMBind, a groundbreaking framework designed to tackle one of the most persistent challenges in multimodal learning: working effectively with incomplete, distributed, and heterogeneous data. In real-world health and behavioral research, collecting complete and synchronized multimodal datasets at scale is often impractical. MMBind offers a powerful solution by enabling robust learning even when modalities are missing or only partially aligned.

The key innovation behind MMBind lies in its ability to “bind” disparate data sources using a sufficiently descriptive shared modality. By leveraging this shared representation, the framework constructs pseudo-paired multimodal datasets that allow models to learn across different modalities—even when observations of the same events are collected at different times, locations, or through different sensors. This capability unlocks new opportunities for analyzing rich, real-world health data that rarely arrives in perfect form.

MMBind also introduces an adaptive multimodal architecture that supports flexible training across heterogeneous modality combinations. To further strengthen performance, the framework integrates a weighted contrastive learning approach, allowing it to better handle domain shifts and variability between datasets.

In evaluations across six real-world multimodal datasets, MMBind consistently and significantly outperformed state-of-the-art baselines under conditions of data incompleteness and domain shift. A paper describing this work has been submitted for publication, marking an important advance in the development of scalable, real-world-ready multimodal machine learning methods for health and beyond.

Paper citation
Ouyang, X., Wu, J., Kimura, T., Lin, Y., Verma, G., Abdelzaher, T., & Srivastava, M. (2024). MMBind: Unleashing the potential of distributed and heterogeneous data for multimodal learning in IoT. arXiv. https://doi.org/10.48550/arXiv.2411.12126