Spoken language in motions: Learning and adaptation of speech communication in the context of body motions (SALAMMBO)

External website: http://salammbo-anr-dfg.ovh

In everyday situations, we often speak while moving (e.g. walking together, cooking, knitting), and move while speaking (e.g. our head, arms and hands, our posture). Breath acts as a mediator between speech and motion and plays an indispensable role in these activities: it provides the brain and muscles with the required oxygen, and the airflow created with exhale is needed for the production of sound. Breathing also sets the pace for the flow of speech: pauses during the inhale are coordinated with prosody and syntax.

Following recent theories on situated and embodied cognition and language, Salammbo’s originality will be to consider limb motion as a common context for spoken language, and breath as a mediator between movement and spoken language. It will adopt an interdisciplinary approach integrating linguistics, motion science and psychology. The first goal is to use advanced technology to create a novel multimodal corpus of simultaneous recordings of limb movement with respiratory, articulatory and acoustic data. A cross-­linguistic longitudinal approach will be adopted. Native speakers of French and German will read and retell stories with novel words on three different days. Idiosyncratic properties of the speakers known to influence breathing and limb movements, i.e. physical fitness and body shape, will be taken into account as determining factors of speech production in the context of body motion. To further assess the speech-­breath-limb link, speech tasks will be performed in different movement conditions of no motion, free hand motion, and rhythmic leg or hand movements.

Based on this corpus, the link between spoken language, breath and limb motion will be addressed in four working packages that analyze: a) the impact of idiosyncratic physical properties on limb motion, respiration and different linguistic levels ranging from phonetics to syntax; b) the impact of limb movement on speech planning, and prosodic and segmental properties of speech; c) the coordination between speech, breath and limb movement using sophisticated time series analyses of synchronizations; and d) the role of limb motion for short-­ and long-­term retention of novel information and vocabulary.

The researchers involved in the two teams have multidisciplinary profiles and complement each other with their expertise in language and cognitive sciences, speech production, multimodal communication, motor control and learning, engineering and signal processing. Different steps have been proposed for spreading the findings to different scientific and clinical communities and to a broader public. The results of the project will not only enrich basic research with a deeper understanding of spoken language in the context of body movement but will also be useful for applied research in language education and speech therapy.