Hand movements

Hand movements (and not especially lip movements, as is often thought); are the best way to recognize language in noisy situations. The best solution is a combination of lip and hand movements. This is the conclusion of a study by a cognitive neuroscientist. A recognizable situation; in a busy café with a lot of background noise, your conversation partner tries to explain something to you.
What do you look at? Are you focusing on their lips or their hand movements? Drijvers It is a bit counterintuitive, but my research shows that hand gestures in noisy situations; contribute more to language comprehension; than just the lip movements. A person’s understanding of language in a difficult listening situation; is greatly enhanced by hand movements, but the best way is to combine lip and hand movements.”

People pronounce verbs

Drijvers had test subjects watch videos in which people pronounce verbs (with a lot of background noise); that express action and make accompanying hand movements. Words like stirring, sweeping, chopping, rubbing, salting and shoving. She looked at what people were looking at (using eye tracking) and how well they understood the verbs with or without hand movements.

Relatively little attention has paid to the visual aspect of spoken language in science, and wrongly so says Drijvers: “Spoken language is usually studied from sound, but the language is not only what you hear, but also what you see. Visual information can make it easier for you to understand the language in difficult listening situations, such as in a noisy café, but also when you have a hearing impairment, or when Dutch is not your native language.”

With the latter group, Drijvers conducted research showing that non-native speakers find it more difficult to recognize speech sounds in noisy environments than native speakers. This is logical in itself, but it also affects the benefits they derive from the information conveying by the lips and the meaningful information conveyed by the hands. As a result, they understand hand and lip movements less well and are worse at linking different information sources. This was reflecting on their behavior; eye movements, and brain activity.

Recognizing lip movements

“For example, in non-native speakers, they noticed less activity in the brain areas involved in recognizing lip movements; brain areas involved in retrieving meaningful information, and brain areas involved in linking speech and hand movements. They were also less able than native speakers to recognize verbs in noisy situations when only lip movements available, and also benefited less from lip movements (and therefore also hand movements), because of they less able to recognize words lost in the noise.”
One scientific side-effect of the research is a new method that can use to investigate the linking of audio-visual information (rapid invisible frequency tagging: RIFT). “This method shows you exactly how information travels through the brain and how it is link in the brain. This provides excellent opportunities for follow-up research into how listeners divide their attention between different sources of information while using language, and how that distribution of attention changes over time. In the end, they will able to understand the role of brainwaves in that process better.”