Multimodality involves the exploration of language use and its representations in human communication, which always draws on the perception of multiple modes. It is not only the auditory channel transmitting acoustic signals, but also the visual channel which contributes to the expression and meaning of an utterance. Speakers use hand gestures and facial expressions to emphasize words or express their feelings and attitudes. We also communicate while on the move. Communication is, therefore, the output of dynamically interacting modes, incorporating acoustic, articulatory, gestural, bodily and respiratory signals. The interplay between these modes is the main focus of our research.

Within such an interdisciplinary approach, we use a large repertoire of methods for data analysis. These include e.g. acoustical analyses, perceptual tests, motion capture data and breathing recordings. Our investigations cover topics on a range of languages such as, for example, German, Polish, French and Turkish. In summary, our overall goal is to better understand the relationships between the modes which create the basis of communication and representation of language.

The resarch theme is supported by:

1. the DFG project Audio-visual prosody of whispered and semi-whispered speech (PROSPER)
2. the DFG project On the Flexibility and Stability of gesture-speech coordination: Evidence from production, comprehension, and imitation (FLESH)
3. the german-polish collaboration project Grammatical tinnitus and its role in the perception of foreign language accent (GRANITUS)
4. the Marie Sklodowska-Curie project Multilingual acquisition and language preservation in Germany: the case of Upper and Lower Sorbian (MAG)
5. the subproject Communicative alignment at the physiological level (ESR 5) of the Marie Skłodowska-Curie Innovative Training Network Conversational Brains (COBRA)

It was also supported by these now completed projects:

1. the german-french collaboration project (ANR/DFG) Spoken language in motions (Salammbo)
2. the DFG project The pragmatic status of iconic meaning in spoken communication: Gestures, ideophones, prosodic modulations (PSIMS) in the framework of the SPP 1727 XPrag.de