ZASxTalk: Multimodal Mobile Lab

Abstract

Interpersonal communication is inherently multimodal. This multimodality is obvious during our face-to-face encounters, where the complex exchanges of auditory, visual and (sometimes) somatosensory information allow us to successfully navigate intricate social situations. Yet, due to the methodological complexity of multimodal research, most empirical investigations into human behaviour have relied on unimodal approaches until not so long ago. Today, the main challenges preventing new developments in multimodal research entail setting-up, collecting and analyzing time varying signals with their own modality-characteristic differences that require careful integration and synchronization.

To address these challenges and foster new developments, we initiated the Mobile Multimodal Lab (MML) project. The MML provides a step-by-step tutorial for building a fully functional multimodal laboratory that relies on open-source resources, integrates synchronized recordings across modalities, remains low in monetary cost, and is both highly modular and easily transportable. It consists of a practical manual and an accompanying coding library supporting the setup, recording, synchronization, and analysis of multimodal data across experimental contexts.

To demonstrate the full MML pipeline, we conducted a proof-of-concept experiment in which two participants engaged in an interpersonal singing task while being simultaneously recorded using six synchronized cameras (for 3D motion tracking), head-mounted microphones (for prosodic analyses), and wireless physiological sensors measuring electrocardiogram, electromyography, and respiration. All data streams were integrated using the Lab Streaming Layer, enabling sub-millisecond synchronization. The complete setup cost less than €10,000 and can be readily extended to include additional measurement systems (e.g., eye-tracking or EEG). The accompanying coding library documents the full experimental pipeline, from synchronization to preprocessing and analysis of multidimensional time-series data. Overall, the MML offers an accessible, open-source framework for capturing the full bandwidth of human behaviors in ecologically valid settings.

Bio

Davide Ahmar is a cognitive neuroscientist interested in the multimodal brain–body mechanisms that support interpersonal coordination and affiliation during real-world social interaction. He completed his BSc in Cognitive Neuroscience and Psychology at the University of Manchester and his MSc in Cognitive Neuroscience at the Donders Institute. During his training, he conducted research at the Italian Institute of Technology on neural and behavioural synchrony, while his MSc research investigated subtle avoidance behaviours in social anxiety, leading to the development of a portable multimodal laboratory.

He is currently a guest researcher at the Max Planck Institute for Evolutionary Anthropology, where he works on visual simulation abilities in great apes, and will soon begin his PhD at the Max Planck Institute for Human Development. In parallel, he is a core member of EnvisionBOX.org, an open-science pedagogical platform for multimodal research, and a co-developer of the Mobile Multimodal Lab project with Dr. Wim Pouw and PhD candidate Šárka Kadavá.

Code of Conduct for ZAS events: The ZAS is committed to fair, respectful, and professional interaction at its events. Therefore, please observe the Code of Conduct for this event.