This project is a project within the DFG Priority Program 2329 - Visual Communication. Theoretical, Empirical and Applied Perspectives (ViCom).

Speech and gestures involve an orchestra of motions working in concert for communicative targets. Several previous studies have shown that speech and gestures are coordinated. However, it is less clear how stable or flexible coordination is and how their coordination evolves. In the FLESH project, we assume that the coordination between speech and gesture is related to the meaning, and plays an important role in the semiotic process. The originality of our approach is that we take the motor properties into account. We suppose that stable gesture-vocal coordination needs to integrate the lighter mass and higher velocities of speech articulators in comparison to arm gestures. Moreover, in contrast to many previous theoretical approaches, we see coordination between speech and gesture as crucial for language evolution. 

To date, gesture studies and language evolution research did not consider motor control in detail and focused on either vocalization or gesture, thus leaving out their coordination and multimodality in general. In the FLESH project, we aim to change this by studying the coordination of vocalizations and gestures in the construction of meaning.

FLESH is organized in four work packages. In work package (WP) 1, we assess the natural coordination tendencies of gesture and vocalization separately and in cooperation when expressing meanings. We assess which coordination tendencies are actually relevant for understanding the meaning of gesture-vocal utterances. In WP 2, state of the art technology and signal processing methodologies will be used to investigate the coordination of the orchestra of multiple motions during imitation. With a high level of granularity, we will be able to study several gesture-vocal articulators that are involved, to see whether particular types of articulators are more likely to coordinate in meaningful versus non-meaningful imitations. In WP 3 we assess how gesture-vocal utterances evolve when they are passed on and imitated by a connected chain of imitators. This experiment will simulate language evolution in the lab. The focus will be on how coordination tendencies change or increase with repetition of gesture-vocal utterances from one chain to the next. In WP 4 we will artificially change the temporal coordination between vocalization and gesture in order to see whether this affects semantic processing. Additionally, gesture-vocal utterances will be presented that do not match in terms of their meaning to see whether novel meanings emerge. The FLESH-project will have an impact on the theoretical understanding of gesture-speech relations, the relation between meaning and motor coordination, and its role in language evolution.