-
Importance of multi-modal communication and how this affects language. 4 additional mechanisms that people use to complement language are...
- gestures
- pauses and disfluencies
- conversation convergence
- speech-vision integration
-
4 types of gestures
- beats - repretitive, simple, no obvious meaning
- pointing
- sybolic (ok and thumbs up)
- lexical gestures: complex, non-repetittive, and related to meaning of accomanying speech
-
What was the old view for gestures?
- Gestures have communicative and informative function
- clear from symbolic gestures and pointing
-
Evidecne against old view of gestures as infomrative?
- However, gestures often tend to be quite vague too and meaningless without words
- Congenitally blind speakers: gesture equally while talking to other blind people
- speakers seem to use it even when they bear no info (Iverson and Goldin-Meadow)
-
What's the more recent view about gestures?
- They facilitate word production
- by facilitating activation of semantic and lexical features
- Butterworth and Hader
- Krauss
-
Evidence for this new view of gestures?
- 1. Blind-to-blind still use gestures (Iverson & Goldin-Meadow)
- 2. Rauscher et al:
- increase disfluencies when gestures prevented
- task - descirbe cartoon. Fake skin conductance electrodes placed on hands so that they won't move them.
- effect esp strong when speech is space-related
- 3. Gesture-speech asychrony: gestures often precede speech
- 500ms-1s before language
- Morrel-Samuels & Krauss (1992)
- 4. Primates and pre-linguistic babies use gestures too (gestures a way of linking facilitating activation of semantic features?)
-
What is the view about evolution of language from gestures?
- Pre-linguistic babies and primates use gestures
- Arbib et al/ Tomasello: language originally conveyed by gestures (getural protolanguage hypothesis)
-
How can pauses and disfluencies help language commincation?
- 1. 60-70% of pauses fall at juncture of sentences (helping with syntactical structure)
- 2. Disfluencies often begin utterance (planning processes - perhaps similar pre-language function to gestures)
- 3. Disfluenceis also often mark production of incorrect word
- 4. Brennan and Schober
- move mouse to purple square but interrupted in different ways
- between-word: 'move to the yellow purple square'
- mid-word: yel purple
- mid-word with filler: yel uh purple
- Fastest response for mid-word with filler
- Disfluency (uh) has some informative purpose
- compensate for disruptions and signal repair
-
The 3rd mechanism - conversation convergence. What is it?
- People in conversation when referring to objects, tend to converge on similar referring expressions. (aka lexical entrainment)
- Metzing and Brennan: confederate speaker say instructions during task requiring moving objects in grid
- Time to look at target slower when old speaker says new word/phrase (eg. silver pipe instead of shiny cylinder)
- But when new speaker says it, no difference in reaction time
- Expect interlocutors to stick to same words/phrases
- encoding who-says what
- distributions of processing load - as each reuses info computed by the other
-
What processes might be underlying conversation convergence?
- Garrod and Pickering (2004)
- automatic mechanisms of priming and imitation
emphasise importance of communicative intent in humans
-
Multimodal integration (with vision).
- McGurk effect
- mouth produce ga, syllable is ba, people report hearing da
- Seeing face supplements noisy signal - raises SNR (signal to noise ratio) by 20dB (Sumby & Pollack)
- See face of speaker enhace comprehension of speech with complicated content or produced with heavy foreign language (Reisberg et al)
-
Evolutionary context of similar multimodal integration.
- Prelinguistic infants link mouth movements to auditory signals they hear (Kuhl & Meltzoff)
- Macaques have highly developed audio-visual integration, which serves to support social integration (Ghazanfar et al)
- Measure single-neuron local field potential in STS (superior temporal sulcus - associative area) and auditory cortex simulteneously
- Function interaction between these stronger when presented with dynamic faces and voices (as opposed to face or voice alone)
- Multisensory neurons in macaque auditory cortex
-
Successful communication in the neural domain.
- Research on neural correlates of communication still in its infancy
- Hassan: communicative interactions supported by coupling of neuronal oscillations (firings) of sender and perceiver (brain-to-brain coupling)
- Difference between stimulus-to-brain coupling and brain-to-brain coupling
- Seeing or hearing will trigger cortical activations in perceiver, leading to coordination and complex joint behaviours.
- Stephens: fMRI of speaker telling story and listener
- Spatially and temporally coupled - (ie the speaker's brain activations modelled that of listener)
- Coupling vanished when Russian recording presented to English speaker, showing it is communicative
- ALSO: frontal regions exhibit predictive anticipatory responses - extent of which correlated with level of comprehension -- suggest prediction is important part of successful communication
|
|