Cued Speech Transliteration: Effects of Accuracy and Lag Time on Message Intelligibility

Abstract

This paper is the second in a series concerned with the level of access afforded to students who use educational interpreters. The first paper (Krause & Tessler, 2016) focused on factors affecting accuracy of messages produced by Cued Speech (CS) transliterators (expression). In this study, factors affecting intelligibility (reception by deaf consumers) of those messages is explored. Results for eight highly skilled receivers of CS showed that (a) overall intelligibility (72%) was higher than average accuracy (61%), (b) accuracy had a large positive effect on intelligibility, accounting for 26% of the variance, and (c) the likelihood that an utterance reached 70% intelligibility as a function of accuracy was sigmoidal, decreasing sharply for accuracy values below 65%. While no direct effect of lag time on intelligibility could be detected, intelligibility was most likely to exceed 70% for utterances produced with lag times between 0.6 and 1.8 s. Differences between CS transliterators suggested sources of transliterator variability (e.g. speechreadability, facial expression, non-manual markers, cueing rate) that are also likely to affect intelligibility and thus warrant further investigation. Such research, along with investigations of other communication options (e.g. American Sign Language, signed English systems, etc.) are important for ensuring accessibility to all students who use educational interpreters.

This paper is the second in a series concerned with the level of access afforded to students who use educational interpreters. Overall, the plan is to examine interpreters in as many communication modes as possible and to focus on clarity of the “visual signals”¹ that the interpreters produce. The rationale is that a clear visual signal from the interpreter is a necessary pre-requisite for deaf and hard-of-hearing students to understand the teacher's message (which ultimately depends on the language used and the student's language processing abilities), in much the same way as a clear auditory signal (i.e. speech) from the teacher is necessary for hearing students to process the message. The goal of the work is to identify the factors responsible for clarity of the visual signal, by examining two channels in the communication pathway: (a) “accuracy”, or the percentage of the message correctly produced by the interpreter, and (b) “intelligibility”, or the percentage of the message that can be correctly received by deaf persons who are proficient in the language and communication mode used by the interpreter.

The first paper in the series (Krause & Tessler, 2016) examined the accuracy of educational interpreters in the United States who use a communication mode known as Cued Speech (CS). Developed by Cornett (1967), CS is a system of manual cues that are produced in synchrony with speech (here, spoken American English), or the visual mouth movements of speech, and designed to disambiguate phonemes confusable through speechreading alone. Initially developed for American English, CS has subsequently been adapted to at least 63 languages and dialects (LaSasso, 2010). Figure 1 shows the CS system for American English, which was the focus of the first paper in the series and remains the focus of this study. It uses eight handshapes to distinguish among 25 consonants and six placements to distinguish among 15 vowels. Handshapes and placements each represent multiple phonemes, but the set of phonemes assigned to a particular cue group are, by design, easily distinguishable via speechreading.

Figure 1

Assignment of consonants to handshapes and vowels to hand placements in Cued Speech for American English. Reprinted from (Krause, Pelley-Lopez, & Tessler, 2011), Copyright 2011, with permission from Elsevier.

Messages produced by CS transliterators² are an attractive candidate for initial study because the phoneme sequence in the spoken message maps directly to the cue sequence in the transliterated message (Krause & Tessler, 2016). Furthermore, it is possible to transcribe the cues produced by CS transliterators with a high degree of reliability, even when the cues are heavily co-articulated (Krause, Pelley-Lopez, & Tessler, 2011). As a result, it is relatively easy to measure CS transliterator accuracy reliably and with a high degree of resolution, using a simple percent-correct metric. This high level of resolution makes it possible to evaluate, quantitatively, the extent to which various factors affect accuracy, which was the focus of the first paper in the series (Krause & Tessler, 2016). With high resolution accuracy measurements for CS transliterators now available, it also becomes possible to determine the extent to which accuracy affects intelligibility of CS transliterators, which is the purpose of this paper.

Intelligibility

Understanding the factors that affect intelligibility is essential for characterizing the level of communication access afforded to deaf individuals by interpreting professionals. Yet to our knowledge, no studies have directly evaluated the intelligibility of any type of interpreter or transliterator, either in ideal conditions or in cases for which the visual signal is degraded. What is known, rather, comes from other methods of assessment that have been used effectively to evaluate the quality of interpreting services. In particular, the Educational Interpreter Performance Assessment, or EIPA (Schick & Williams, 1994) has proven to be a valid and reliable (Schick, Williams, & Bolster, 1999) research tool for examining the quality of educational interpreters, with studies using the EIPA having reported data for more than 2,000 educational interpreters nationwide (Schick, Williams, & Kupermintz, 2006). Designed to assess the performance of interpreters who use American Sign Language (ASL), Manually Coded English systems, Pidgin Signed English, and more recently, Cued Speech (EIPA-CS; Krause, Kegl, & Schick, 2008), the EIPA examines a large number of specific interpreting skills (organized into four skill domains) using a Likert scale ranging from zero (no skill) to five (advanced skill). While the Likert scale is well-chosen for the purposes of the EIPA, a metric with a higher degree of resolution is required for investigating the quantitative effects of various factors on intelligibility (in order to ensure that small effects can be detected). Moreover, it would be more convenient for such analyses if this metric were focused solely on intelligibility. Therefore, a percent-correct measure of intelligibility is used in this study.

Although no previous studies have used percent-correct intelligibility measures to evaluate CS transliterators or other types of interpreters, several studies have reported using this intelligibility evaluation procedure for situations involving direction communication in CS. Such studies have shown, for example, that when materials are perfectly cued, experienced users of CS can obtain near-perfect reception of words (Nicholls & Ling, 1982) and everyday sentences (Uchanski et al., 1994), while for sentences with low context, reception is somewhat lower, with reports ranging from 84% (Uchanski et al., 1994) to 91% (Duchnowski et al., 2000) intelligibility on average. While it seems reasonable to expect similar intelligibility levels for transliterated messages that are perfectly cued, accuracy of CS transliterators is often less than perfect (Krause & Tessler, 2016). Thus, intelligibility of transliterated messages must be measured directly, and the effect of factors such as accuracy and lag time on intelligibility must also be evaluated.

Accuracy

One aspect of the visual signal that is very likely to affect intelligibility is accuracy. Although there is no known quantitative research regarding the accuracy of most types of interpreters, some information regarding the accuracy of transliterators³ who use CS is available. In the first paper of this series, Krause and Tessler (2016) reported an average accuracy of 54% for 12 CS transliterators (with varying experience levels, averaged over three different rates of presentation), with 24% of target cues omitted, and 22% produced in error (i.e. substitutions, or incorrect cues). Insertions of cues accounted for an extraneous 5% beyond the expected target cues. Although the average accuracy was quite low, it is worth noting that higher accuracy was observed in a number of different circumstances. At the slow speaking rate, for example, accuracy was 64% on average and as high as 89% for one particular CS transliterator.

At the phrase level, a wide range of accuracy scores was observed, ranging from near 0% to near 100% (Krause & Tessler, 2016). Such accuracy measurements, however, do not measure directly how accessible each phrase would be to a deaf receiver with fluent English skills who is proficient in CS. That is, it cannot be assumed that transmitting 75% of the message faithfully renders it 75% intelligible. Rather, it is possible—indeed, likely—that intelligibility is somewhat higher than accuracy. Given that CS requires a transliterator to mouth the message while producing the accompanying hand cues, it could be that deaf consumers can still recognize some words even when errors are present via a combination of speechreading and of extracting information from any cues that are correct. Also, for certain type of errors (e.g. those predictably related to the underlying correct cues and produced relatively consistently), it may be possible for deaf consumers to compensate for or adjust to a CS transliterator's style of cueing, much like a listener can adjust to a talker's accent. Moreover, the relationship between accuracy and intelligibility is not necessarily linear; in speech, for example, the relationship between physical properties of the stimulus and intelligibility is often sigmoidal, or S-shaped (e.g. Wilson & Strouse, 1999). Whether a similar relationship would hold for the relationship between accuracy and intelligibility of visual messages is unknown. To answer this question, the effect of accuracy on intelligibility must be examined.

Lag time

Another factor that could affect message intelligibility—at least indirectly—is lag time, or the average delay (in seconds) between the spoken message and the transliterated message. Although the relationship between intelligibility and lag time has not been explored in any modality, an inverse relationship between accuracy and lag time has been established, both for American Sign Language interpreters (Cokely, 1986) and for CS transliterators (Krause & Tessler, 2016). However, the relationship for CS transliterators was quite weak accounted for just 3% of the variance in accuracy. Whether this weak inverse relationship is preserved in the relationship between lag time and intelligibility or obscured by other factors is not yet known. In either case, it may be possible to identify a lag time or range of lag times that optimize intelligibility. Consistent with Cokely's hypothesis that lag time is proportional to structural differences between the source and target languages, average lag times for CS transliterators (1.86 s; Krause & Tessler, 2016) are typically shorter than those for the ASL interpreters (3 s; Cokely, 1986); however, it is unknown if these lag times are optimal for intelligibility for either group. Therefore, research is needed to determine the nature of the relationship between intelligibility and lag time.

Present study

In the present study, eight highly skilled receivers of CS were presented with visual stimuli excised from transliterated messages produced for the earlier study (Krause & Tessler, 2016) on CS transliterator accuracy. Receivers were asked to transcribe the stimuli, and intelligibility was measured as the percentage of words correctly received. Two characteristics of the visual signal were derived from previous measurements of the transliterated messages: (a) accuracy, measured as a percent-correct score based on the target cue sequence, and (b) lag time (in seconds), reported as the average delay between the spoken message and the transliterated message. The goal of the experiment was to determine the effect of accuracy and lag time on intelligibility levels.

Method

Participants

In order to evaluate the intelligibility of the cued materials described below, eight adults who were highly skilled receivers of CS for American English were recruited via announcements to email lists, social media groups, and organizational newsletters (e.g. National Cued Speech Association) that target the CS community. All (five females, three males; age range: 20–39 years) reported English as a first language, possessed at least a high school education, and had no known visual acuity problems. To qualify as a “highly skilled” CS receiver, each participant (CS-RO1–CS-R08) was required to meet the following criteria: (a) introduced to CS before age 10, (b) used CS receptively (or receptively and expressively) at home (with at least one parent) and at school (through a teacher or CS transliterator) before age 18, and (c) had at least 10 years of experience using CS. In addition, participants were required to pass a receptive CS proficiency screening. The screening consisted of five conversational English sentences obtained from a list of Clarke sentences (Magner, 1972) that were presented via CS alone (i.e. no audio). The sentences were cued with 100% accuracy, and participants were required to transcribe, verbatim, 90% or more of the words correctly in order to pass the screening.

Because the experimental format relied upon written English responses from participants, the Expressive Written Vocabulary section of the Test of Adolescent and Adult Language Third Edition (TOAL-3; Hammil, Brown, Larsen, and Wiederholt, 1994) was also used to screen for basic proficiency in written English. Participants were required to score within one standard deviation of age-appropriate averages. Because TOAL-3 normative data does not include deaf and hard-of-hearing individuals, this screening procedure ensured that all participants, regardless of level of hearing loss, possessed an English proficiency level on par with typical high school graduates. All eight participants who were recruited for the study met this criterion and also passed the receptive CS proficiency screening.

Finally, each participant completed a survey regarding communication background and level of hearing loss. Table 1 summarizes the information collected from this survey and from the two screening tools for each of the eight participants.

Table 1

Language, education, and communication background of participants

	CS-R01	CS-R02	CS-R03	CS-R04	CS-R05	CS-R06	CS-R07	CS-R08
Age (yrs)	27	30	39	34	25	28	20	33
Gender	M	F	M	F	M	F	F	F
Education	B.A.	Some college	B.A.	PhD	Some college	Some college	Some college	B.A.
Hearing loss	Profound	Profound	Profound	Severe to Profound	Profound	Profound	Profound	Profound
CS receptive screening (%)	100	100	100	100	100	100	100	95
TOAL-3 percentile	95th	95th	98th	84th	75th	75th	84th	75th
First Language	English	English	English	English and French	English	English	English and ASL	English
Age (yrs) first CS exposure	2	5.5	3	2	2	0.83	1	3
CS home use (during childhood)	Both parents, always	Mother only, sometimes	Both parents, always	Mother—always, Father—sometimes	Both parents, always	Both parents - always before implanted at age 8, moderate amount after	Mother—always, Father—only for younger years	Both parents, always
CS school use (during childhood)	K-12	K-6	K-12	Some in middle school	K-12	1–12	1–12	K-12
CS experience (yrs)	25	24	36	32	23	27	19	30
Preferred Comm. Mode	English (oral and cued)	No response	English (oral and cued)	Spoken English ASL	Cued Speech	Cued Speech	Cued Speech/English	Cued Speech and Sign Language
Fluency—other (age of initial exposure, yrs)	Signed English (18)	None	None	French (2), ASL (17)	ASL (17)	Sign Language (19)	ASL (1), Spanish (12)	None

Materials

Intelligibility stimuli were generated from cued materials recorded for an earlier study on the factors affecting accuracy of 12 CS transliterators (CST01–CST12; Krause & Tessler, 2016). In that study, the 12 participants were asked to transliterate an audio lecture about plants presented at three different speaking rates: a slow-conversational rate of 88 words-per-minute (wpm), a normal-conversational rate of 109 wpm, and a fast-conversational rate of 137 wpm. The lecture was drawn from a 25-min educational film designed for use in a high school setting, entitled “Life Cycle of Plants” (Films for the Humanities, 1989). The film provides an overview of key topics in plant growth and reproduction and includes some specialized vocabulary pertaining to plants (e.g. names of plant species). In order to create a lecture version of the material, the audio narration from the film was re-recorded by a male talker who read a transcript of the film, using a lecture-style delivery with deliberate pauses at phrase boundaries. Each CS transliterator was then presented with this lecture in three segments (one segment per speaking rate), counterbalanced across speaking rates in order to minimize any accuracy effects that might be due to differences in difficulty of segments (see Krause & Tessler, 2016, for details).

Using these materials, the goal for the current study was to extract intelligibility stimuli with a wide range of accuracy scores produced at a single speaking rate so that the effect of various levels of accuracy on intelligibility could be examined independent of speaking rate. Therefore, all intelligibility stimuli were drawn from materials elicited at the slow-conversational speaking rate, because accuracy data from the first paper in this series (Krause & Tessler, 2016) showed that CS transliterators at this speaking rate had exhibited a wide range in accuracy, with phrase level scores ranging from 0% to 100%. As this speaking rate, materials were available from four different CS transliterators for each of the three lecture segments (Segment 1: CST01, CST02, CST07, CST08; Segment 2: CST03, CST04, CST09, CST10; Segment 3: CST05, CST06, CST11, CST12). Thus, materials were available from all 12 CS transliterators, with each sentence of the lecture produced by four different transliterators.

Stimulus preparation

In order to prepare materials that were appropriate for use in the intelligibility experiment, video clips consisting of short utterances were excised from the recordings of each CS transliterator, using Adobe Premiere Pro 1.5, resulting in roughly 75 video clips per CS transliterator. The utterances were typically segmented at natural phrase boundaries that were marked by pauses in the audio narration. In some cases, however, the anticipated break point was not available, either because the CS transliterator did not pause as expected or because the cues produced by the CS transliterator led or lagged the corresponding mouthshapes by such a degree as to obscure the break point. In these cases, two consecutive phrases or short sentences were either combined or divided at alternate break points, provided that the modified utterances remained semantically appropriate and did not contain more than 12 words.⁴ This upper limit on length was imposed in order to limit each utterance to a manageable number of bits of unrelated information such that participants would be able to remember the exact utterance (Miller, 1956) long enough to transcribe it.

Prior to stimulus selection, three properties of the utterance produced in each video clip were documented: (a) lag time, (b) accuracy, and (c) average key word accuracy. Both lag time and accuracy were either previously measured or easily derived from data that was previously collected for an earlier study on the relationship between accuracy and lag time (Krause & Tessler, 2016); key word accuracy was calculated by computing the average accuracy for each key word, then averaging the word-level accuracies of all key words in the utterance. Key words were identified by a panel of experts in sign transliteration as content words that are required for full comprehension of the meaning of the sentence by deaf consumers (Kile, 2005).

Stimulus selection

In all, roughly 900 video clips elicited at the slow-conversational rate were available for use in this study (3 lecture sections × 4 CS transliterators per section × ~75 phrases per section/transliterator). These video clips included approximately four instances (one per CS transliterator) of each of the roughly 225 phrases in the audio narration. In order to evaluate the intelligibility of as many of these clips as possible, four unique stimulus sets (i.e. groups of video clips) were created, with each set arranged such that all phrases from the audio narration could be presented to the receiver in order. The four stimulus sets were counterbalanced across receivers in order to minimize stimulus-specific effects and the effects of other factors on the accuracy-intelligibility relationship.

In selecting the stimuli for each stimulus set, the primary goal was to obtain a wide range in accuracy scores, with a relatively uniform distribution from 0% to 100%, so that the relationship between accuracy and intelligibility could be assessed across a full range of accuracy scores. Similarly, clips with a variety of lag times were also selected, so that the effect of different lag times on intelligibility could be evaluated. Figures 2 and 3 (top panel), respectively, show that these goals were achieved in each of the four stimulus sets. In addition, all 12 CS transliterators were represented in relatively equal proportions within and across stimulus sets, as shown in the lower panel of Figure 3. Finally, a secondary goal of stimulus selection was that whenever possible, clips with similar accuracy scores or with similar lag times be distributed to different locations in the lecture.

Figure 2

Distribution of accuracy and key word accuracy scores for stimuli in each of the four stimulus sets.

Figure 3

Top panel shows distribution of lag time values for stimuli in each of the four stimulus sets. Number of stimuli are shown for each 0.5 s range of lag times (i.e. 0.5 represents lag times greater than zero and less than or equal to 0.5 s). Lower panel shows number of stimuli representing each CS transliterator across the four stimulus sets. CS = Cued Speech.

Because each stimulus set was required to contain all phrases from the narration in order, there were some constraints on availability of clips with particular criteria. Most notably, fewer clips were available at some accuracy levels (especially in the 0–35% accuracy range) than others. Similarly, there were limitations on the number of clips available from each CS transliterator because some phrases were omitted by one or more transliterators. Available lag time values were characteristic of CS transliterator behaviors and could not be manipulated. As a result of these constraints, it was necessary to select some clips for more than one stimulus set; specifically, 98 clips were used in two stimulus sets, 37 clips were used in three sets, and 47 clips were used in all four stimulus sets. The remaining 405 stimuli selected were unique to a single stimulus set, which allowed for intelligibility evaluation of a total of 587 video clips.

Procedures

All participants were tested individually at a computer in a sound-treated room at the University of South Florida. Presentation sessions were conducted in two 2-hr sessions, with one 15-min and two 10-min breaks per session. Participants were also encouraged to take breaks as necessary to maximize attention and reduce any possible fatigue effects.

Instructions to participants were given both in written English and in spoken English with CS as needed. After completing the English language and receptive CS screenings at the first session, participants were administered a practice stimulus set in order to become familiar with the experimental setup and procedures. During the practice list, participants were given the opportunity to ask procedural questions as needed. Once the practice list was completed, the experimental stimuli were presented.

Stimulus presentation

For context, stimulus items were preceded by short scenes from the source material for the transliterated lecture an educational film, “Life Cycle of Plants”. Scenes from the film were presented in order, and at the conclusion of each scene, one or more stimulus items were presented corresponding to the audio narration for that scene. Both the scene and the subsequent stimulus item(s) were visual-only and contained no audio. Each stimulus item consisted of one phrase (i.e. video clip) of the transliterated message, which participants were asked to transcribe, verbatim, by typing into a response box. Participants controlled the rate of presentation of the stimuli via a customized user interface implemented in MATLAB (Mathworks, 2007) software and could thus pause for as long as needed to complete each response. However, they were only permitted to view each video (i.e. scene from the film or stimulus item) one time.

Given the goals of stimulus selection, consecutive stimulus items were not necessarily produced by the same CS transliterator. However, each of the three lecture sections contained materials from just four CS transliterators, which afforded participants a chance to become familiar with the transliterators throughout the course of a lecture section.

Subjective ratings

After each section of the lecture was completed, participants were asked to rate each of the four CS transliterators from that section by completing a short survey. With pictures of the CS transliterators for reference, the survey required participants to select which CS transliterator they perceived as most effective and which one they perceived as least effective for that section. In addition, participants were instructed to rate how they would feel about using each of the CS transliterators in a real-life setting by choosing from three possible ratings: “Very comfortable,” “OK,” or “Concerned I might miss something.” Participants were also given the option to comment about anything they particularly liked or disliked about each CS transliterator.

Scoring

Intelligibility scoring included analysis of two types of intelligibility: “original message” (OM) intelligibility, or the percentage of the original spoken message that was correctly received by the participant, and “transliterated message” (TM) intelligibility, or the percentage correctly received of just that portion of the message that was actually cued by the transliterator. OM intelligibility was used to provide an overall measure of intelligibility that captures how much access deaf receivers actually receive from a transliterated lecture, while TM intelligibility was used to focus on the intelligibility of an individual CS transliterator's message, even if that message differed from the original message (either because the transliterator rephrased the material or omitted words or sequences of words).

Percent-correct scores were obtained by tabulating the amount of agreement between each of the typed responses and the corresponding source message (for OM intelligibility) or message that was transliterated (for TM intelligibility). Three types of scores were calculated for each response: (a) original message (OM)—all words, (b) original message (OM)—key words, and (c) transliterator message (TM)—key words. In all types of scoring, credit was given for obvious spelling and typographical errors as well as homophonous words (e.g. “mail” for “male”). For key word grading, the experimenter also gave credit for morphological errors that involved the addition or deletion of an affix (e.g. “running” for “run” or “ran”) as well as contractions when both words were target words (e.g. “don't” for “do not”) and vice versa. Finally, because many of the proper nouns and names of plant species were difficult to spell, the correct criteria were loosened slightly for these words. Specifically, these words were simply required to sound phonetically similar to the target word, with all word scoring also requiring that the number of syllables match the target word.

Results

Table 2 summarizes the physical characteristics (accuracy, key word accuracy, and lag time) of the stimuli presented in the experiment, as well as the corresponding intelligibility results (OM—All Word, OM—Key Word, and TM—Key Word) for each participant, on average. The information regarding physical characteristics confirms that the stimuli selected for all four stimulus sets (viewed by two participants each) were well-balanced across participants with respect to accuracy scores, key word accuracy scores, and lag times. That is, each participant viewed a stimulus set that averaged 61% accuracy across all stimuli presented. The average key word accuracy of the stimuli was also approximately the same for each participant, with overall key word averages of 75% and 76% for each participant's stimulus set. Lastly, the average lag time of the stimuli presented to participants varied only slightly, with average lag times per stimulus set ranging from 1.8 s to 1.98 s, a difference of less than 10%.

Table 2

Stimulus characteristics and intelligibility by participant (averaged over stimulus set)

Participant	Characteristics of stimuli received			Intelligibility
	Accuracy (%)	Key Word Accuracy (%)	Lag Time (s)	OM-All Word (%)	OM-Key Word (%)	TM-Key Word (%)
CS-R01	61	75	1.98	79	83	89
CS-R02	61	76	1.80	72	78	83
CS-R03	61	75	1.98	76	84	89
CS-R04	61	76	1.80	72	79	84
CS-R05	61	75	1.84	69	75	80
CS-R06	61	75	1.82	74	76	81
CS-R07	61	75	1.84	68	70	76
CS-R08	61	75	1.82	65	70	74
Average	61	75	1.86	72	77	82
Range	8–100	0–100	0.34–7.31	0–100	0–100	0–100

The overall intelligibility for these stimuli (averaged across all CS transliterators and all participants) was 72% for all words in the original message, 77% for key words in the original message, and 82% for key words in the transliterated message. Although differences in absolute performance levels were observed, with scores varying by 14–15 percentage points within conditions (e.g. in the OM All Word condition, CS-R08 had the lowest average intelligibility of 65% while CS-R01 had the highest average intelligibility of 79%), the relative intelligibility of these three measures was consistent across individual participants. That is, for all participants (regardless of their absolute performance levels), intelligibility scores were highest for the TM—Key Word measure and lowest for the OM—All Word measure (OM All Word < OM Key Word < TM Key Word).

A comparison of the average intelligibility to average accuracy also revealed a consistent pattern. The OM—All Word intelligibility score obtained by each participant was higher—considerably higher, in most cases—than the accuracy average of the stimuli; the advantage was 11 points on average (72% vs. 61%) and ranged from 4 to 18 points across individual participants. This advantage appears, at least in part, due to the fact that key words were produced with somewhat higher accuracy than other words in the original message (i.e. Key Word Accuracy > Accuracy). In other words, OM—Key Word intelligibility and key word accuracy were very similar (77% vs. 75% on average), with six of eight participants obtaining an OM—Key Word intelligibility score that was within 5 points of the average key word accuracy of stimuli (75%).

Individual CS Transliterator Results

As Table 3 shows, data for individual CS transliterators followed the same pattern as the overall intelligibility results: TM intelligibility scores were the highest of the three intelligibility measures, followed by OM—Key Word intelligibility, and finally, OM—All Word intelligibility. Another similarity was that intelligibility scores were substantially higher than accuracy for most of the individual CS transliterators; the advantage ranged from 5 to 23 percentage points for 8 of 12 transliterators. Interestingly, the 3 CS transliterators with the highest accuracy (CST03, CST07, CST12) were not able to obtain intelligibility scores that were higher than accuracy levels, suggesting that a CS transliterator's accuracy and intelligibility may merge as accuracy reaches the highest levels. CST01 was the only CS transliterator with accuracy below 80% who was not able to obtain an intelligibility score that was higher than her accuracy. The large difference between OM and TM intelligibility scores suggest a high amount of paraphrasing used by this CS transliterator, which is likely obscuring the effect by lowering her OM intelligibility.

Table 3

Stimulus characteristics and intelligibility by Cued Speech transliterator (averaged over stimulus set)

Transliterator	Characteristics of selected stimuli			Intelligibility
	Accuracy (%)	Key Word Accuracy (%)	Lag Time (s)	OM-All Word (%)	OM-Key Word (%)	TM-Key Word (%)
CST01	68	70	3.30	66	66	95
CST02	67	81	1.66	75	79	79
CST03	81	85	3.42	76	82	86
CST04	40	59	3.41	54	66	79
CST05	69	85	1.92	77	86	88
CST06	71	84	1.10	86	88	88
CST07	86	95	1.13	87	91	91
CST08	51	75	1.21	74	78	78
CST09	59	75	1.10	76	79	79
CST10	47	67	0.76	52	60	60
CST11	73	85	1.41	90	94	93
CST12	90	96	1.10	87	91	91

Although the CS transliterator results were generally consistent with the overall results, the range of absolute intelligibility scores amongst individual CS transliterators was much greater than what was observed amongst participants. Intelligibility in the OM—All Word condition, for example, ranged 38 percentage points, from 52% (CST10) to 90% (CST11). Given that the accuracy averages of stimuli selected from each CS transliterator were also quite different (see Table 3), an examination of the CS transliterators’ average intelligibility scores provides some insight into the relationship between accuracy and intelligibility.

As expected, the CS transliterators’ intelligibility scores generally followed accuracy, with CS transliterators who had the higher accuracy averages also obtaining higher intelligibility scores and CS transliterators with lower accuracy averages obtaining lower intelligibility scores. This pattern may be easiest to see by comparing the rank order of CS transliterators based on accuracy averages with the rank order of CS transliterators based on average intelligibility, as seen in Table 4. An inspection of these rank orderings reveals that, for the most part, accuracy and intelligibility rankings are very similar. For example, CST12 is ranked highest of the CS transliterators in accuracy and second highest in intelligibility, while CST04, on the other hand, is ranked lowest (12th) and second lowest (11th) in these categories, respectively. In a few cases, however, accuracy rankings for each CS transliterator are somewhat different than intelligibility rankings; specifically, CST01 and CST03 both ranked lower in intelligibility than accuracy, while CST09 and CST11 both ranked higher. The two who ranked lower, CST01 and CST03, were two of the four CS transliterators whose accuracy averages were nearly the same as their intelligibility scores. As a result, they were each outranked in intelligibility by CS transliterators (three transliterators in the case of CST01 and four transliterators in the case of CST03) whose intelligibility exceeded accuracy (as was the case for the majority of CS transliterators). Notably, this type of outranking did not happen to the other two CS transliterators in this situation (CST07 and CST12), perhaps because their accuracy and intelligibility were approaching ceiling. As for CST09 and CST11, the two CS transliterators who ranked considerably higher in intelligibility than accuracy, the reason for their unexpected highly intelligibility is unknown and suggests that there are factors beyond accuracy that can be used to improve CS transliterator intelligibility.

Table 4

Cued Speech transliterator rankings by accuracy, intelligibility, and subjective ratings

Ranking	Accuracy	OM All Word intelligibility	Subjective ratings
1st	CST12 (90%)	CST11 (90%)	CST07 (9.375)
2nd	CST07 (86%)	CST12 (87%)	CST11 (8.75)
3rd	CST03 (81%)	CST07 (87%)	CST12 (6.875)
4th	CST11 (73%)	CST06 (86%)	CST03 (5)
5th	CST06 (71%)	CST05 (77%)	CST06 (4.375)
6th	CST05 (69%)	CST09 (76%)	CST05 (3.75)
7th	CST01 (68%)	CST03 (76%)	CST01 (2.5)
8th	CST02 (67%)	CST02 (75%)	CST02 (2.5)
9th	CST09 (59%)	CST08 (74%)	CST09 (1.875)
10th	CST08 (51%)	CST01 (66%)	CST04 (1.25)
11th	CST10 (47%)	CST04 (54%)	CST08 (0)
12th	CST04 (40%)	CST10 (52%)	CST10 (0)

In addition to the CS transliterator rankings for accuracy and intelligibility, Table 4 includes CS transliterator rankings derived from participants’ subjective ratings. To obtain the rankings, point values were assigned to each participant's responses (1.25 points for each “Very Comfortable” rating, 0.625 points for each “Okay” rating, and 0 points for each “Concerned” rating), yielding a composite rating from 0 (when all eight receivers rated the CS transliterator with “Concerned”) to 10 (when all eight receivers rated the CS transliterator with “Very Comfortable”). For the eight CS transliterators who obtained similar rankings in both accuracy and intelligibility, rankings based on subjective ratings were also similar to these two rankings. Of the remaining four CS transliterators, three had subjective rankings that were similar to accuracy rankings (including CST01 and CST03 who ranked lower on intelligibility than accuracy and CST09 who ranked higher on intelligibility than accuracy). However, the subjective ranking of CST11 (who ranked first in intelligibility despite ranking fourth in accuracy) was most similar to his/her intelligibility ranking.

Effect of Accuracy on Intelligibility

In order to examine more closely the relationship between CS transliterator accuracy and message intelligibility, scatterplots relating the accuracy and intelligibility of individual stimulus items for three combinations of measures were examined: (a) accuracy versus OM—All Word intelligibility, (b) accuracy versus OM—Key Word intelligibility, and (c) key word accuracy versus OM—Key Word intelligibility. Because results were similar in all three cases, only the relationship between accuracy and OM—All Word intelligibility, shown in Figure 4, will be discussed here. As expected, there was a positive relationship between accuracy and intelligibility; a Spearman's rank order test of correlation confirmed the relationship was moderate in strength and highly significant (Spearman's rho = 0.478, p < 0.0005).⁵ In addition, the variation in accuracy accounted for 26% of the variation in OM—All Word intelligibility scores. In other words, the linear fitting function plotted in Figure 4 provides a reasonably good prediction, on average, of how intelligibility changes with accuracy.

Figure 4

Relationship between accuracy and intelligibility of individual stimulus items (open circles represents each instance of a stimulus item) for OM-All Word intelligibility.

For individual stimulus items, however, estimating intelligibility from the linear fitting function is not always appropriate. As Figure 4 shows, the range of intelligibility scores corresponding to a particular accuracy is generally quite large; as a result, the intelligibility of some stimulus items is quite far from the intelligibility predicted by the linear fitting function. Another issue is that the intelligibility scores of individual stimulus items are not normally distributed around the fitting function. The density of data points in Figure 4 shows that for many accuracies (e.g. 60% and higher), the most frequently occurring intelligibility score is 100%, rather than the intelligibility score predicted by the linear fitting function. Thus, for predicting the intelligibility of individual stimulus items, a function which characterizes the likelihood of a particular intelligibility score at a given accuracy may have more practical value.

Toward this end, Figure 5 shows the likelihood that, for a stimulus item with a particular accuracy, participants would be able to obtain an intelligibility score of at least 70%. The likelihood function was determined by calculating the proportion of individual stimulus items that had greater than 70% intelligibility within each 10-point accuracy interval. Thus, it approximates the probability that a particular participant would receive 70% or more of the words in a particular stimulus item correctly, for any stimulus item in the interval. As such, the likelihood values provide some indication of the distribution of intelligibility scores for each accuracy interval. Figure 5 shows that for the data in this study, the accuracy-intelligibility likelihood function appears roughly sigmoidal in shape, showing relatively little change in intelligibility likelihood at the ends of the accuracy scale (0–20% accuracy and 65–100% accuracy) and a steeper slope in the middle of the scale where increases in accuracy cause more dramatic increases in intelligibility likelihood. In other words, the likelihood of reaching 70% intelligibility drops off very quickly as accuracy falls below 65%.

Figure 5

Proportion of data with >70% intelligibility as a function of accuracy. Each filled circle represents the proportion of data points that reach 70% or higher intelligibility scores within each 10-point range in accuracy. Accuracy and intelligibility of individual stimulus items (open circles) are also plotted for reference.

Effect of Lag Time on Intelligibility

Figure 6 shows a scatterplot relating the lag time and OM—All Word intelligibility of individual stimulus items. Although no linear relationship between the two variables was detected (Spearman's rho = −0.041, p = 0.319), the density of data points suggested that higher intelligibility scores occurred more frequently for shorter lag times. To investigate this possibility, an intelligibility likelihood function, representing the proportion of stimulus items with intelligibility scores greater than 70%, was calculated using lag time intervals of 0.6 s.

Figure 6

Likelihood that individual stimulus items have greater than 70% intelligibility (OM-All Word) as a function of lag time. Each filled circle represents the proportion of data points that reach 70% or higher intelligibility scores within each 0.6-s range in lag times. Lag time and intelligibility of individual stimulus items (open circles) are also plotted for reference. OM = original message.

The shape of the lag time-intelligibility likelihood function confirms that intelligibility scores of greater than 70% were most likely to occur for lag times between 0.6 and 1.8 s (i.e. lag times associated with the 0.6-s intervals centered around 0.9 and 1.5 s, respectively). Of the stimulus items with lag times in this range, a relatively high proportion (0.73 to 0.78) were associated with intelligibility scores of greater than 70%; therefore, this range can be considered an optimal lag time for CS transliterators, at least for the materials and speaking rate used in this study. As lag time increased beyond 1.8 s, the decline in intelligibility likelihood was initially somewhat steep, decreasing from 0.78 for the interval centered on 1.5 s to only 0.54 for the interval centered on 2.1 s. However, intelligibility likelihood did not decrease substantially beyond this point; for stimulus items with longer lag times (>2.4 s), likelihood of reaching at least 70% intelligibility ranged from 0.43 to 0.62. The lowest intelligibility likelihood, 0.33, occurred for stimulus items with lag times that were shorter than those in the optimal lag time range (i.e. less than 0.6 s), suggesting that very short lag times are more likely to be detrimental to intelligibility than longer lag times.

Discussion

The results of this study show that message intelligibility is 72% of all words in the original message, on average, for 12 CS transliterators conveying educational materials designed for high school settings to highly skilled CS receivers. Intelligibility is higher for key words in the original message (77% on average) and highest for key words in the transliterated message (82% on average). The relatively high TM—Key Word intelligibility scores (74–89%)⁶ demonstrate reasonably good CS reception by participants, despite varying degrees of transliterator errors (not all words attempted by CS transliterators were cued with 100% accuracy). Yet even this level of intelligibility is still well below the analogous measure obtained from normal hearing listeners (when presented with audio stimulus items of 100% accuracy), who received, on average, 98% of all words in the original message and 99% of key words (Tope, 2008). Nonetheless, it is encouraging that (a) intelligibility is higher than accuracy on average (72% vs. 61%) and for most individual CS transliterators, and (b) TM—Key Word intelligibility was quite high for some CS transliterators (e.g. 95% for CST01) and higher than OM—All Word intelligibility for every CS transliterator. Together, these two points suggest that if CS transliterator accuracy were to increase, intelligibility of the original message could be improved.

Of course, the ultimate goal of successful communication is more than intelligibility; it is comprehension of the message. For deaf students in classroom settings, recent research suggests that comprehension and retention may be affected by method of delivery, at least in some circumstances; that is, deaf students who received direct instruction in ASL comprehended more content from a science lesson than those who received the lesson via a highly skilled ASL interpreter (Kurz, Schick, & Hauser, 2015). It is as yet unknown whether this finding (a) applies more broadly to deaf students who use CS or other English-based communication modes, and/or (b) reflects underlying differences in intelligibility between the two methods of delivery. In any case, it underscores the importance of systematically examining comprehension, intelligibility, and the factors that affect them both.

In this study, the primary purpose was to examine the effects of two such factors, accuracy and lag time, on intelligibility of messages produced by CS transliterators. Of these two factors, lag time had the smaller effect on intelligibility. Despite a wide variation in lag times, no linear relationship between lag time and intelligibility could be detected. However, an examination of the intelligibility of individual stimulus items suggests an optimal range of lag times between 0.6 and 1.8 s; in this range, intelligibility scores greater than 70% occurred most frequently. The increased likelihood for relatively high intelligibility over this range of lag times may be explained, in part, by accuracy. Given that there is a weak negative relationship between accuracy and lag time (Krause & Tessler, 2016), stimulus items with lag times in the optimal range are likely to have somewhat higher accuracy (on average) and thus somewhat higher intelligibility than those with lag times longer than 1.8 s. For items with very short lag times (i.e. less than 0.6 s), however, it is unclear what role accuracy could play, if any, in reducing intelligibility likelihood.

What is clear is that accuracy as an independent factor had a fairly large effect on intelligibility. As hypothesized, the relationship between accuracy and intelligibility was positive; as accuracy increased, intelligibility increased. The linear relationship between the two variables was statistically significant, with accuracy accounting for 26% of the variance in CS transliterator intelligibility. Moreover, it is worth noting that this is a conservative estimate; it is possible that somewhat more variance could be explained with refinements to how accuracy and intelligibility measurements are made. The accuracy measurements used in this study were taken at cue level (the basic unit of CS, consisting of a handshape and a placement representing one consonant-vowel unit), while intelligibility was scored at the level of whole words. If both were measured in the same manner, it could be that accuracy may account for somewhat more than 26% of the variance in intelligibility.

Regardless, a large portion of variance remains unexplained by accuracy or lag time alone, or even from the two together. As Figure 5 shows, for example, a particular accuracy is associated with a wide range of intelligibility scores. While the variability can be seen at all accuracies, it is particularly striking at the upper end of the scale. That is, even an utterance produced with 100% accuracy is not guaranteed to achieve 100% intelligibility. For the materials in this study, just 63% of stimulus items produced with 100% accuracy were received by participants with 100% intelligibility, and while the vast majority (94%) were at least 70% intelligible, even this level of intelligibility was not a guarantee—in one case, intelligibility was as low as 29%. This variability has at least two important scientific implications. First, intelligibility cannot be judged or predicted from accuracy alone; rather, it is more appropriate to estimate intelligibility likelihood. Second, sources of the remaining variability should be explored in order to better understand all of the factors that can affect intelligibility. In addition, there are numerous practical implications that should be considered by the CS transliteration community. Each of these topics is discussed in more detail below.

Intelligibility Likelihood

As discussed previously, the linear relationship between accuracy and intelligibility is of limited practical value in predicting the intelligibility of individual stimulus items. This limitation is a result of two issues: (a) the range of intelligibility scores associated with a particular accuracy is generally quite large, and (b) the intelligibility scores are not typically normally distributed across the range. An intelligibility likelihood function, which calculates the proportion of individual stimulus items that meets or exceeds a minimum intelligibility threshold (e.g. 70%), overcomes these limitations to some extent by taking into account the distribution of intelligibility scores associated with changes in a physical property of the stimulus (e.g. accuracy). In this study, the intelligibility likelihood calculated as a function of accuracy was sigmoidal in shape, with the likelihood of reaching 70% intelligibility dropping off very quickly as accuracy falls below 65%.

The shape of the intelligibility likelihood function underscores the importance of CS transliterators producing messages in the upper accuracy range where intelligibility likelihood is relatively stable; operating at lower accuracies greatly reduces the reliability of message transmission. In this sense, the accuracy-intelligibility likelihood function can be likened to psychometric functions that have been used to document the influence of various factors on speech reception (e.g. Wilson & Strouse, 1999); in so doing, it must be noted that the specific properties of the sigmoidal shape—that is, the slope of the middle portion of the function and the left-right shift (i.e. the range of accuracies corresponding to the precipitous drop-off)—are likely to reflect the conditions of this experiment. Therefore, it should not be assumed that 65% accuracy would always be associated with an intelligibility likelihood comparable to higher accuracies; rather, the knee point (accuracy associated with the point in the curve where intelligibility begins to level off) may change for situations with different speech materials (i.e. different content areas and different language levels), speaking rates, etc.

Inasmuch as the accuracy-intelligibility likelihood function is indeed like a psychometric function, it may then have great potential as a research tool for assessing the effect of different variables on the accuracy-intelligibility relationship—not only in messages produced by CS transliterators, but in all types of interpreted messages. For example, the minimum accuracy necessary to achieve a particular level of intelligibility likelihood may vary with content area (i.e. subject matter of the transliterated message) and/or speaking rate; this possibility could be assessed by comparing the slope and left-right shift of this function for materials across a range of content areas presented to transliterators/interpreters (within a particular language or communication mode) at different speaking rates. Similarly, this type of analysis could be used to examine differences in individual transliterators (or interpreters) or in individual receivers. Such an analysis could lead to identifying traits of particularly effective interpreting professionals, or perhaps even suggest how interpreting professionals and receivers can be matched most effectively. In this study, for example, the resulting data might provide more insight into participants’ subjective ratings of CS transliterators. Although the ratings (described above) were generally in good agreement, there was at least one instance which shows how widely participant assessments can diverge. Specifically, CST12 was selected by 6 out of 8 participants as “highly effective,” but one participant found her to be “highly ineffective.” Perhaps an inspection of the individual accuracy-intelligibility likelihood function for this participant processing stimulus items produced by CST12 might reveal a very different shape that could help explain the ineffective rating. Unfortunately, such an analysis is beyond the scope of this study, due to the limited number of stimulus items (roughly 20, depending on the stimulus block and CS transliterator accuracies available) each participant was presented from a given CS transliterator.

While it was possible to construct accuracy-intelligibility likelihood functions for individual CS transliterators in this study by pooling data across all participants, in most cases the function could not be fully characterized because stimulus items from some CS transliterators were restricted in accuracy range. For example, the range of accuracies utilized as stimulus items for CST12 was restricted to higher accuracy values only, allowing only the rightmost portion of the function to be characterized (Pelley, 2008). In contrast, accuracy-intelligibility likelihood functions of individual participants (i.e. receivers) could be constructed across a wide range of accuracies by pooling data across all CS transliterators. A preliminary analysis suggests these functions are generally similar in shape, with differences mostly in left-right shift indicating differences in absolute performance levels between participants (Pelley, 2008). However, given the high degree of variability (i.e. range of intelligibility scores corresponding to a given accuracy), more data is needed to confirm the sigmoidal nature of the relationship in each case.

Sources of Variability

The large amount of variance in intelligibility that remains unexplained after accounting for the effects of accuracy and lag time suggests other sources of variability are likely to play a role in intelligibility. The most obvious sources that are likely to affect intelligibility stem either from differences in CS transliterators or differences in receiver characteristics.

CS transliterator variability

Possible sources of CS transliterator variability are numerous and include the transliterator's cueing mechanics (handshape formation, placements, and transitional cueing movements), speechreadability, error types, facial expressions, cueing rate, synchronization between mouth and cues, and prosody (i.e. timing and emphasis). Other sources of CS transliterator variability may include intra-transliterator factors (e.g. fatigue, attention, nervousness), differences in cue selection (e.g. cueing unstressed /i/, as in “funny” as /i/ versus /I/), and dialect (dialect may be apparent in selection of cues as well as mouth movements). Of these possibilities, research on displays for automatic cueing systems suggest that transitional cueing movements and synchronization between mouth and cues are both likely to play a significant role (Krause, Duchnowski, & Braida, 2010). Speechreadability is also expected to have a large contribution, because mouth movements are an integral part of the CS signal, and speechreadability is known to vary both within talkers (Jiang, Auer, Abeer, Keating, & Bernstein, 2007) and across talkers (Kricos & Lesner, 1982, 1985).

From this study, some insight into these and other factors can be gained by examining CS transliterators who had sizeable differences in their accuracy and intelligibility rankings. CST11, for example, ranked fourth in accuracy at 73%, but achieved the highest intelligibility (90%) of any CS transliterator, surpassing even CST12 (87% intelligibility), the CS transliterator with the highest accuracy (90%). Although the accuracy difference between these two CS transliterators might be smaller if CST11 were awarded partial credit (i.e. half-credit for substitution errors that included either a correctly produced handshape or placement) for her tendency to “hypocue” (approximate, but not fully achieve the intended handshape or placement), it would still not be possible to explain the intelligibility difference between the two CS transliterators by accuracy alone. That is, even if CST11 were to receive partial credit for all of her substitution errors while CST12 received none (a highly unlikely scenario), CST11's accuracy would rise only to 84%, which would still fall short of CST12 at 90%. Therefore, CST11 must have capitalized on other skills to achieve such high intelligibility. A close inspection of CST11's performance reveals several strengths which may have played a role: (a) high speechreadability, (b) excellent facial expressions and other non-manual information (use of eyebrows, head movements, and the non-cueing hand) to show questions, emphasize important points, show lists, and convey the tone of the message, and (c) effective use of available time while still keeping up with the message (i.e. using speaker pauses to slow down slightly in order to better show syllable, word, or sentence stress).

At the other end of the continuum are CST01 and CST03, the two CS transliterators whose intelligibility was worse than expected given their accuracy scores. Categorized as “novice” CS transliterators by Krause and Tessler (2016), these transliterators had much less experience than the rest of the group, which may have been a contributing factor. Apart from experience level, however, specific factors likely to have negatively impacted intelligibility that were observed in both of these two CS transliterators include slow cueing rate, misleading facial expressions (concentrating, confused, or discouraged facial expressions), and poor timing and rhythm within words and sentences (extraneous pausing, poor representation of syllable and word stress). CST03 also held her hand at an atypical angle and had many false starts. These false starts, combined with extraneous pausing, resulted in a misleading rhythm that may have obscured word boundaries at times (in one case, a participant perceived “surgically precise” as “surge eucalyptus”). Slow cueing rates for both CS transliterators also resulted in average lag times (CST01: 3.4 s; CST03: 3.5 s) much longer than the optimal lag time range of 0.6–1.8 s. Furthermore, it is possible that the cueing rates were too slow at times and may have caused participants to have difficulty keeping words in working memory. All or most of these transliteration behaviors are likely to reflect the novice CS transliterators’ difficulty coping with the cognitive load and/or physical demands of the task of transliterating at a conversational speed, and at least some of them are likely to be detrimental to intelligibility.

Finally, discrepancies between the intelligibility and subjective ratings for one CS transliterator (CST08) may also be an indication of additional sources of CS transliterator variability. Based on subjective ratings, CST08 ranked last (tied with CST10; both received a composite rating of 0 on a scale of 0–10) and was unanimously identified as “highly ineffective,” despite achieving 74% intelligibility (ranking 9th of 12). Several transliteration behaviors were noted as possible factors that may explain why the subjective ratings for this CS transliterator were poorer than would be expected based on intelligibility alone. First, when producing cues that would be expected at the chin, side, and throat placements, CST08's hand frequently remained in front of the chin, making it difficult to distinguish the intended placement of these cues. Many of the participants complained about this aspect of CST08's cueing, both on the survey and in conversations at the breaks during the experiment. Second, CST08 regularly used unusual and sometimes misleading mouthshapes, frequently producing lip rounding in words that should not contain it (e.g. “slightest”). Next, CST08 exhibited poorer synchronization between mouth and cues than other CS transliterators; frequently, her mouth was still articulating the sound(s) of the previous sentence while beginning to cue a word from the next sentence or vice versa. Her extremely low subjective rating suggests either a conscious level of awareness of one or more of these shortcomings or a more general frustration on the part of participants when receiving messages produced by CST08. While two other CS transliterators with higher subjective ratings were less intelligible (CST01 and CST04), it is likely that their cues were clearer. If CST08's transliteration left participants with more uncertainty regarding what cues had been produced, participants’ cognitive load may have been increased. A higher cognitive load would not only increase fatigue but also receiver variability.

Receiver variability

Possible sources of receiver variability (other than increased cognitive load) include communication background as well as experience and comfort level with receptive cueing. Because deaf individuals encounter more diverse communicative environments than do hearing individuals, these individuals are likely to have less consistent exposure to their chosen communication modes than hearing individuals. While at least one participant (CS-R01) reported using cueing all the time with most members of his/her family, several of the participants in this study reported that they are now (in their adult lives) primarily in non-cueing environments, either with hearing people who do not cue or with deaf people in settings where sign language is the primary mode of communication (for work or with deaf friends/spouses who do not cue). At least one participant (CS-R03) reported feeling “rusty” with receptive CS, saying s/he relied heavily on speechreading during the testing. It seems likely that receivers who are less experienced with receptive cueing, or those who have with reduced comfort levels due to lack of recent use, are likely to experience increased intelligibility variability.

Regardless of comfort level, there may also be differences between cue receivers in processing strategies used for cue reception. Some receivers may rely more heavily on information from mouth movements (and prefer CS transliterators who exhibit a high degree of mouthshape clarity), while others may rely more heavily on information from the cues (and prefer CS transliterators who exhibit a high degree of cue clarity). The former group would be expected to make errors that are more frequently consistent with the mouthshapes but incongruent with the cues produced, while the latter group would be expected to do the reverse (more frequently making errors that are consistent with the cues but incongruent with the mouthshapes produced). Of course, for all receivers, some of both error types may occur, and both types of errors were indeed noted in this study. An initial inspection of participant responses suggests that the overwhelming majority of errors made by the eight participants in this study were more consistent with mouthshapes than with target cues; however, it is unknown whether errors in cue production (made by the CS transliterator) may have influenced these errors. While this question could be answered to some extent by aligning cue-by-cue production accuracy data with receiver responses, such an undertaking was beyond the scope of this study. And in any case, research with a substantially larger group of receivers would be needed in order to determine the number and types of processing strategies used by consumers.

Implications for Practitioners

While the field of interpreting cannot control for differences in receiver performance, it should be possible to improve overall intelligibility by reducing sources of CS transliterator variability. If, for example, the contribution of CS transliterator factors likely to affect intelligibility (speechreadability, facial expressions, cueing rate, etc.) can be quantified, a hierarchy of important skills could be developed, which would aid in training and assessment of CS transliterators. Based on what is currently known, CS transliterator training should place substantial importance on the accuracy of the cues, while also giving some attention to lag time; but because these factors alone do not guarantee 100% intelligibility, it is also important that some training be dedicated to other factors expected to increase intelligibility, such as those discussed above.

When assessing CS transliterators, the results of this study show that it is important (at least until the effect of additional CS transliterator factors can be adequately understood) to evaluate intelligibility (through informal and formal assessments by highly skilled receivers of CS) as well as accuracy whenever possible. Moreover, this assessment should be undertaken with materials that are comparable (in speed, content area, and difficulty) to materials in the CS transliterator's work environment, because the accuracy-intelligibility likelihood function is expected to change with these factors. For assessments wherein intelligibility evaluation is not practical (e.g. live educational settings), it is imperative to evaluate a multiplicity of factors (e.g. speechreadability, non-manual markers, prosodic information, contextual indicators, etc.) alongside accuracy; accuracy alone is just one important contributor to intelligibility. Another important point is that even when CS transliterators are at their best, operating in the upper accuracy range where intelligibility likelihood is high and relatively stable, intelligibility of difficult material (like the materials used in this study) can still vary from utterance to utterance. Thus, repetition (upon consumer signal/request) may sometimes be necessary to maintain message clarity in these situations.

Lastly, while materials in this study did not include sufficient numbers of CS transliterators to make generalizations regarding the role of experience and certification, a few points are worth noting. First, there was a general tendency for increased experience level to be associated with increased intelligibility. In particular, those with the least experience (CST01, CST03; Krause & Tessler, 2016) achieved lower intelligibility scores and rankings than would be predicted from the performance of their more experienced counterparts. Nevertheless, more experience did not always guarantee higher intelligibility; the two CS transliterators with the lowest intelligibilities (CST04, CST10) were among the three most experienced transliterators in the group. Second, while CS transliterators holding state certification varied widely in intelligibility, the two who held national certification (CST07, CST12; Krause & Tessler, 2016) achieved very high intelligibility at 87%. While this level of performance suggests possible differences in certification rigor between national certification and state certification (at least in some states), data from more CS transliterators is needed before any firm conclusions can be drawn. Finally, it should be acknowledged that lack of certification status does not necessarily mean poor intelligibility; the most intelligible CS transliterator in the study, CST11, did not hold certification of any kind (Krause & Tessler, 2016).

Conclusions

Although additional research is needed, the results of the present study are an important first step toward quantifying factors that affect the message intelligibility of CS transliterators. Of the two factors examined, accuracy had the larger effect on intelligibility, accounting for 26% of the variance. For most CS transliterators, average intelligibility exceeded average accuracy, and the differential was quite large in some cases (ranging from 5 to 23 points). Lag time had no direct effect on intelligibility, but intelligibility scores greater than 70% occurred most frequently for lag times ranging from 0.6 to 1.8 s, suggesting that this range is most likely to be associated with optimal intelligibility. Although the role of these two variables in intelligibility is substantial, 74% of the variance in intelligibility remains unexplained (and as a result, wide variations in intelligibility scores were evident at any given accuracy), which suggests other sources of variability beyond those examined in this study are likely to play a role in intelligibility. While some sources of variability may be random or attributable to receiver characteristics (which are difficult to control), those sources of variability associated with CS transliterators are important to understand because they can potentially be adjusted by transliterators to improve intelligibility.

Sources of CS transliterator variability identified in this study point to a number of factors (e.g. speechreadability, facial expression, non-manual markers, and cueing rate) that may influence intelligibility and thus warrant further investigation. Specifically, future research should quantify the contribution of each of these factors by evaluating the extent to which each one affects intelligibility. This information could then be used to improve the efficacy and efficiency of training and assessment, perhaps by introducing procedures that weight each factor according to its relative importance. In addition, more research is needed examining additional CS transliterators at every experience level, from a wide variety of backgrounds (i.e. various credentials, amounts and types of training, etc.) in order to determine the relationship between more general CS transliterator characteristics (e.g. experience, level of certification) and CS transliterator intelligibility. Finally, similar experiments are needed for transliterators and interpreters who use communication modes other than CS (e.g. American Sign Language, Conceptually Accurate Signed English, Signing Exact English, etc.). Such research, alongside research examining factors affecting message comprehension, would increase our understanding of what accuracy levels, and other factors, are needed by interpreting professionals in each modality in order to ensure accessibility to the teacher's message and other classroom communication.

Notes

1

The term “visual signal” used here is a shortened version of Battison's (1978) observation that sign language is “a manually produced, visually received signal” while speech is “an orally produced, auditorily received signal.”

2

Whereas the function of an interpreter is to translate between two languages (e.g. spoken English and ASL), the function of a transliterator is to transfer information between two modes of the same language (e.g. spoken English and either signed English, cued English, or speechreadable English). Thus, we use Cued Speech transliterator here and throughout the article to describe the work of any interpreting professional who transfers spoken English information to/from Cued Speech. We occasionally employ the phrase “interpreters who use Cued Speech” when our goal is to reference Cued Speech transliterators within the context of the interpreting profession as a whole (Krause, Kegl, & Schick, 2008, p. 432).

3

The term “transliterator” may be used as a shortened form of “Cued Speech transliterator.” For clarity, however, we will do so sparingly in this article—and only when the full meaning can be determined easily from context. Our primary usage of “transliterator” will be as a more general term referring to any interpreting professional who transliterates (i.e. signed English transliterators, Cued Speech transliterators, oral transliterators, etc.).

4

For two transliterators, one 13-word phrase was allowed because no other appropriate break points could be found. However, the phrase appeared to have no more than seven independent, meaningful chunks of information, suggesting a likelihood that most participants would be able to remember it relatively well for a short amount of time (Miller, 1956).

5

This non-parametric test of correlation was employed because the data are not normally distributed; as is visible on the graphs, the distribution of the accuracy-intelligibility functions is skewed toward 100%, given that a substantial number of stimuli reached the maximum intelligibility values.

6

TM intelligibility here is averaged across transliterators and would be higher for some individual transliterators, presumably those with the highest accuracy.

Funding

Financial support for this work was provided in part by a grant from the National Institute on Deafness and Other Communication Disorders (National Institutes of Health grant number 5 R03 DC 007355).

Conflicts of Interest

No conflicts of interest were reported.

Acknowledgments

The authors wish to thank Wendy Fuchs and Jane Smart for assistance with stimulus preparation, Kendall Tope Beaudry and John Lum for assistance in stimulus creation, and Danielle Milanese for donated transliteration services used to develop practice items. In addition, we thank Catherine Rogers for helpful technical discussions throughout the process. We also thank Joe Frisbie for the cue chart used in Figure 1.