Individual variation in speech perception and production

Individual variation in speech
perception and production
Valerie Hazan
Dept of Phonetics and Linguistics
UCL, London, UK

Aim? To understand this!
Why do these productions sound so
different?
Why do these vary in intelligibility for
different listeners?
What is the link between acoustic
aspects of the productions and their
perception?

Why study variation?
Inherent part of ‘real’ communication
To build more robust models of speech
perception and production
To build better practical applications:
Speech technology – ASR and synthesis
Speech and language therapy
Language training

Why is this becoming a ‘hot
topic’?
Move away from ‘laboratory speech’ to use of
more natural speech stimuli
Individual variability no longer dismissed as
‘experimental noise’
Emergence of models of speech perception
such as hyper-hypo theory, exemplar model..
Increasing number of speech tech and other
applications

Complexity, complexity
everywhere…
Main problem in studies of variation? Unpicking various sources of
variation
Speaker-related variability
Cross-speaker
Anatomical differences (vocal folds, vocal tract)
Accent
Individual articulatory behaviours (e.g. identical twin
studies..)
…
Within-speaker
Speaking style
Speech rate
Physical and mental health
…
Listener-related variability
Auditory processing abilities
Perceptual weighting of acoustic cues
Relative use of acoustic and linguistic info

Does variability in speech production
affect speech perception?
Efficient process of normalisation but
Speakers not equally intelligible
Different styles of speech not equally
intelligible
Variability might affect perception in
impoverished listening conditions
(‘stressed’ system)
Some listener populations may be less able
to cope with variation than others?…

Does variability in speech production
affect speech perception?
Let’s remember that poor acousticphonetic
information can be
compensated by efficient use of
contextual/linguistic information
BUT greater cognitive load when
redundancy of information is reduced?

An example of the ‘power’ of
context…

Some key topics
Speaker-related variability
To what degree do speakers vary from each other
in terms of how intelligible they are?
How much of the variation is due to anatomical
differences?
Are child speakers more variable than adult
speakers?
Listener-related variability
Do listeners vary in the acoustic-phonetic info they
use to decode speech?
Do children vary from adults in the use of this
information?

Some methodological issues..
Two approaches in studies of speaker
‘clarity’ and variation:
‘Deliberately-clear’ speech: comparison of
different elicited speaking styles within
speakers
‘Intrinsically-clear’ speech: comparison of
different speakers using ‘set’ style (usually
read speech)

Part 1: Speaker-related
variation and its impact on
intelligibility

A study of effect of speaker-related variability
on speech intelligibility - Hazan and Markham
(2004, 2005)
To what degree do speakers vary from each other
in terms of how intelligible they are for a fixed set
of materials?
Is speaker intelligibility consistent across listeners
differing in age and sex?
Can we find correlations between speaker
intelligibility and acoustic-phonetic characteristics
of their speech?

UCL Speaker Database (Markham
& Hazan, 2002)
Speakers
o
o
45 speakers (18 women, 15 men, 6 girls and 6 boys
aged 12)
Homogeneous accent group (South-eastern British
English accent)
Listeners
135 listeners (same accent group)
45 children aged 7-8,
45 children aged 11-12
45 adults

UCL Speaker Database
Word level
Nonsense VCV lists
Markham word test
Manchester Junior word lists
Sentence level
Semantically-unpredictable sentences (SUS)
Accent revealing sentences
Read passages
2 text passages (Arthur, Rainbow)
Semi-spontaneous monologues
Description of cartoon
Retelling of cartoon without picture

Study 1: Intelligibility tests
Markham Word test
Set of 124 monosyllabic words:
o familiar to children > 7
o cover all frequent consonant confusions, with
large spread of vowels
o E.g. might, net, night, sat, shop…

Study 1: Intelligibility tests
Methodology
- Listeners tested in quiet classroom via
headphones
- Oral response given to experimenter
- In the triplet condition, each listener heard 25
unique words from each of 15 speakers,
presented in a fully randomised fashion.

A little test…’good’ or ‘poor’
speaker?
Ranked 1st out of 45
Ranked 9 th out of 45
Ranked 45th out of 45
Ranked 2nd out of 45
Ranked 44th out of 45

Error rate (%)
SPEAKER
af-06
af-14
am-10
am-08
af-21
af-02
af-12
am-07
cm-04
am-19
af-10
cf-01
af-09
cf-04
cm-05
af-13
af-11
af-16
am-05
af-19
af-04
cm-02
am-09
am-02
af-18
af-17
am-06
cf-06
cf-08
am-03
am-18
cm-01
am-16
am-17
af-08
cm-03
af-07
cf-03
cf-09
am-12
af-15
cm-06
am-13
af-03
am-14
0
10
20
across speakers?
What is the range in intelligibility rates

Intelligibility for speaker groups
differing in sex and age
1.00
0.95
0.90
0.85
0.80
30
29
Adult listeners
Women Men Girls Boys
ANOVA on
arcsinetransformed
intelligibility
rates
Women are
more intelligible
than other
speaker groups
No difference in
intelligibility
between groups
of children and
men

Intelligibility rates for
individual speakers
Note degree
of variability
within each
gender/age
group
Women Men girls boys

Speaker intelligibility:
Cross-listener correlations
Word Intelligibility - OC listeners (Prop.)
1.00
.90
.80
.70
.70
.80
.90
1.00
Word Intelligibility - YC Listeners (Prop.)
1.00
.90
.80
.70
.70
.80
.90
1.00
Word Intelligibility - YC listeners (Prop.)
1.00
.90
.80
.70
.70
.80
.90
1.00
Talker Group
Boys
Girls
Men
Women
Total Population
Rsq = 0.8562
Word Intelligibility - Adult listeners (Prop.)
Word Intelligibility - Adult Listeners (Prop.)
Word Intelligibility - OC Listeners (Prop.)
Remarkable consistency in relative speaker intelligibility
across the three listener groups:
Pearson’s correlation: r=0.90 to r=0.95

What does this tell us?
Speakers differ significantly in intelligibility
even for single-word materials presented in
low-level noise
There is a tendency for women to be more
intelligible than men or children.
There is remarkable consistency across
listener groups as to which voices are more
or less intelligible
So intelligibility of a voice is based on
something inherent to the speaker rather
than on relation between speaker and listener

Some shortcomings…
Results based on ‘laboratory speech’
(single words read in anechoic
chamber)
How do these speakers sound in
connected speech?
‘Best’ speaker
‘Worst’ speaker

Can we understand why some
speakers are more intelligible than
others?
Approach?
Analysis of acoustic-phonetic
characteristics of speech stimuli
Correlation with intelligibility results

What do previous studies tell
us?
Cross-speaker studies
Bond and Moore (1994): speech rate, vowel
space,
Bradlow et al (1996): fundamental frequency
range, vowel space measures, precision of
articulation
Speaking style studies
Picheny (1985, 86): speech rate, articulatory
precision
Krause and Braida (2004): energy in midfrequency
range, low-frequency modulation in
intensity envelope

Acoustic-phonetic correlates
of intelligibility?
The following measures were made:
Long-term average spectrum (based on all words)
Slope and energy in 1-3 kHz frequency band
Fundamental frequency (based on all words)
F0 mean, range, open quotient, irregularity
Speaking rate (based on subset of words)
Measure of mean word duration
CV amplitude ratio (based on subset of words)
CV ratio for fricative-vowel, stop-vowel, nasal-vowel
Formant spacing (based on subset of words)
Euclidian distance F1/F2 for /i, A, u/, difference in F1
between /i/ and /a/ (in erb), difference in F2 between /i/ and
/u/ (in erb).

Significant correlation between intelligibility
and total energy in 1-3 kHz region
.20
.18
Mean word error rate (prop.)
.16
.14
.12
.10
.08
.06
.04
Speakers
Children
Rsq = 0.1395
Men
Rsq = 0.5655
Women
Rsq = 0.5161
Total Population
.02
Rsq = 0.4036
-10
-8
-6
-4
-2
0
2
4
Total energy 1-3 kHz

Significant correlation between
intelligibility and word duration (speech
rate)
.20
.18
.16
Word error rate (prop.)
.14
.12
.10
.08
.06
Speakers
Children
Rsq = 0.0049
Men
Rsq = 0.4163
Women
Rsq = 0.1202
.04
Total Population
.02
Rsq = 0.1284
.3
.4
.5
.6
.7
Mean word duration (sec.)

Significant if weak correlation between
intelligibility and measure of vowel space
.20
.18
.16
Word error rate (prop.)
.14
.12
.10
.08
.06
Speakers
Children
Rsq = 0.0016
Men
Rsq = 0.5912
Women
Rsq = 0.2162
.04
Total Population
.02
Rsq = 0.1552
1.0
2.0
3.0
4.0
5.0
6.0
Difference in F2 (ERB) between /i/ and /u/

BUT lots of variability…
Profiles of ‘best’ and ‘worst’ speakers
Talker Rank (1-45)
Word
Intel. (all)
LTAS
1-3 kHz
Word duration Diff. F2 /i/-
/u/
af-06 1 1.5 3 6
af-14 2= 19= 16 11
am-10 2= 9 17 40
am-08 4 27 25 18
af-12 5 35 1 16
. . . .
am-12 40 42 39 42
af-15 41 39 29 39
cm-06 42 19= 19 4
am-13 44 43= 40 44
am-14 45 39 44 45

Conclusions
Is speaker ‘clarity’ inherent to the speaker or might listeners
find voices closer to their own easier to understand?
Intelligibility appeared to be determined by inherent
characteristics of their speech rather than by an interaction of
listener and speaker-related factors…Implications for
exemplar-based theories?
Are intelligibility rates correlated with acoustic-phonetic
measures?
YES: Correlation with energy in mid-frequency band,
speaking rate and vowel space BUT strength of correlation
varies with speaker groups. No single measure was
necessary or sufficient to produce highly-intelligible speech. It
seems likely that high intelligibility in individual speakers
arises from different combinations of acoustic-phonetic
features.

Why do acoustic-phonetic
correlates vary across studies?
Methodological differences
Studies of ‘intrinsically-clear’ vs ‘deliberately-clear’
speech
Types of materials used: e.g. fundamental
frequency measures more likely to be important in
connected sentences?
Lack of strong correlation between any one
acoustic-phonetic measure and intelligibility
Strong correlations would imply that different
listeners rely on the same acoustic-phonetic
information..is this the case?

Are we barking up the wrong
tree?
Maybe what’s important is not how
specific acoustic-phonetic characteristics
are produced by a speaker but..
the degree of ‘internal consistency’ in
the production of phones (overlapping
or non-overlapping
categories)…(Newman et al, 2000)

Newman et al. (2001) data
• Slower reaction
times for fricatives
for speaker
showing overlap in
/s/-/S/ distributions
• Reflects greater
‘processing load’?

Part 2: Listener-related
variation

A bit of history…
Early studies of speech perception
Main issues: What acoustic cues are
primary for perception of phonemic
contrasts?
Presentation of mean identification or
discrimination functions
No discussion of individual differences in
identification ability or cue weighting
‘Poor’ performers sometimes eliminated from
dataset

Early models of speech
perception
Search for invariant correlates of
linguistic units
Assumption that variability which is
irrelevant for decoding of speech is
‘stripped away’
Motor Theory: invariant units are
articulatory gestures
Quantal Theory: invariant units are
acoustic properties of the signal

Why an increasing interest in
variability?
Vocal tract normalisation theories vs
exemplar-based memory models
‘Rather than warp the input signal to match a
fixed internal template, the internal
representation adapts according to the
“perceived identity of the talker”’ (Johnson,
1990)

Individual differences in
speech processing
Clearly present in individuals with hearing
loss but not immediately evident in normallyhearing
subjects…
Why?
Speech is highly redundant so individuals are
functioning ‘at ceiling’
BUT individual differences are obvious when
the system is ‘stressed’ (e.g. speech in noise,
simultaneous tasks)

Where might individual
differences arise?
Auditory processing
Processing of acoustic-phonetic
information (acoustic cues)
Use of contextual/linguistic information

What about listener-related
differences in our study of word
perception in noise?
What was the range in intelligibility across listeners?
.20
Adult listeners
.18
Mean error rate (triplets)
.16
.14
.12
.10
.08
.06
.04
.02
0.00
1
4
7
10
13
16
19
22
25
28
31
34
37
40
43
subject number

Are there individual differences in
auditory processing?
Study of individual differences in
processing of speech and non-speech
sounds (Surprenant and Watson, 2001)
Group of 93 subjects tested on:
Speech perception tasks (syllables, words,
sentences in noise)
Spectral-temporal discrimination tasks
using simple and complex nonspeech tasks

Individual differences in
auditory processing tasks
From Surprenant and Watson, JASA, 110, 2001

Individual differences in
sentence in noise tasks
Psychometric function for
sentences in noise for
subjects divided in group of
ten. Only groups
1,2,3,7,9,10 are shown.
(Surprenant and Watson,
2001)

Conclusions of study
Significant individual differences in auditory
processing
Significant individual differences in ‘normal
population’ for speech perception in noise:
E.g. at SNR of –1.6 dB, best 10% recognise 82%
of words whilst worst 10% recognise 38% of
words.
Little correlation between performance on
auditory and speech tasks (but ‘global’ vs
‘analytic’ tasks?)

Are there individual differences in
the processing of acoustic cues?
What about individual differences in the
ability to extract phonetic information
from the signal (phonemic
categorisation tasks)?

Let’s think more about the
notion of ‘categorisation’
?
L
R

Phoneme categorisation task -
Example
‘speech continuum’
Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7

What tasks do you carry out
with this continuum?
Identification task
- Sounds presented in
random order (many
repetitions)
- Each time, listener says
which sound was heard
(e.g. R or L)
% R
Percentage "Date" responses
100
Identification function
80
60
40
20
0
1 2 3 4 5 6
Stimulus steps
gradient of function
phoneme boundary

Individual differences in the use of
acoustic cues (Hazan et al, 1990s)
Contrasts:
GATE-DATE, GAT-DAT, GEET-DEET
GATE-KATE, GAT-KAT, GEET-KEET
Test conditions
combined-cue and ‘single cue’ in order to
evaluate ‘weighting’ given to each acoustic
cue.

An example:
DATE-GATE endpoints
Burst and F2 varying
Only F2 varying
Only burst varying

Listeners & Test procedure
Listeners
50 adults (mean: 21.1 yrs) with normal pure tone
thresholds
Native speakers of British English
Test procedure
Two-alternative forced choice identification task
32 responses per stimulus per listener collected
over four sessions
Combined-cue and single-cue stimuli randomised
together

d/-/g/ contrast: individual
differences in cue-weighting
There was little variability in cue-weighting in the context of /i/, but in
the context of /ei/ or /a/, there was greater variability in the use of
burst or transition information. Some listeners focused on a specific
cue, others required cue redundancy whilst a small number were able
to make use of whichever information was present.
Percentage of listeners
affected by cue removal
100%
80%
60%
40%
20%
0%
GATE-DATE GAT-DAT GEET-DEET
Minimal Pair
No burst-cue
No transitions-cue
Both
Neither

g/-/k/ contrast: Individual
differences in cue-weighting
Less than 5% of listeners were using the F1 transition cue in the
context of /ei/ and /i/. However, in the context of /ae/, even though on
average the effect of F1 transition cue removal was not significant,
over 30% of listeners were significantly affected by the absence of this
cue.
Voicing contrast: Effect of cue-removal
% listeners affected by
cue-removal
100%
80%
60%
40%
20%
0%
GATE-
KATE
GAT-
KAT
GEET-
KEET
No F1 transition cue
CONTRAST

Discussion
Acoustic cues vary in their “robustness”: much less
individual variability in the use of the clearly dominant
VOT cue to voicing than in the use of burst/formant cues
to the place contrast.
Individual differences in cue-weighting strategies used by
listeners. For the /d/-/g/ contrasts, some listeners
appeared to focus on a specific cue (either burst or
formant transition information), whilst a small number
appeared to be flexible in their use of whichever cue was
present.

Are there individual differences in the
use of contextual cues? – one
example
Not surprising that listeners vary also in
the way they make use of linguistic
contextual information
One useful test: SPIN (Kalikow et al.,
1977).

SPIN test
HP The watchdog gave a warning growl.
HP She made the bed with clean sheets.
LP The old man discussed the dive.
LP Bob heard Paul call about the strips.
Tells us how well people can make use of
semantic information, but tells us less about the
use of phonetic information

Some data showing variation in the
effect of contextual information
55 adult listeners
tested on 50 SPIN
sentences
100
80
SPIN (% correct)
60
40
Low predictability
20
High predictability
SUBJECT

So?
Evidence of individual variability at all
levels of perception
Auditory processing
Processing of acoustic-phonetic cues
(‘analytic’ tasks)
Use of contextual information

My main message for today!
Complex interaction between variability
in production and perception
Due to individual variability in production,
acoustic-phonetic cues present in the
signal will vary from speaker to speaker.
Some listeners seem to be more flexible in
their use of acoustic cues and can cope
with impoverished acoustic-phonetic info,
others less so.

Bibliography
Bond, Z., & Moore, T. (1994). A note on the acoustic-phonetic characteristics of
inadvertently clear speech. Speech Communication, 14, 325-337.
Bradlow, A., Torretta, G., & Pisoni, D. (1996). Intelligibility of normal speech .1. global and
fine-grained acoustic-phonetic talker characteristics. Speech Communication, 20, 255-272.
Hazan, V. and Markham, D (2004) Acoustic-phonetic correlates of talker intelligibility for
adults and children. JASA, 116, 3108-3118
Kalikow, D., Stevens, K. and Elliot, L. (1977). Development of a test of speech intelligibility
in noise using sentence materials with controlled word predictability. Journal of the Audio
Engineering Society, 19, 920-922.
Markham, D. and Hazan, V. (2002). UCL Speaker Database. Speech, Hearing and
Language: UCL work in progress, vol. 14, 1-17.
Markham, D. and Hazan, V. (2004) The effect of talker- and listener-related factors on
intelligibility for a real-word, open-set perception test. Journal of Speech, Hearing and
Language Research, 47, 725-737.
Newman, R., Clouse, S., & Burnham, J. (2001). The perceptual consequences of
withintalker variability in fricative production. JASA, 109, 1181-1196.
Picheny M.A., Durlach N.I., Braida L.D. (1985), Speaking clearly for the hard of hearing .1.
intelligibility differences between clear and conversational speech, J. Speech Hear. Res. 28,
96-103.
Surprenant, A.M. & Watson, C.S. (2001) Individual differences in the processing of speech
and nonspeech sounds by normal-hearing listeners. JASA, 110, 2085-2095.