Signing Exact English Transliteration: Effects of Speaking Rate and Lag Time on Production Accuracy

Abstract

This paper is the third in a series concerned with the level of access provided to deaf and hard of hearing children who rely on interpreters to access classroom communication. The first two papers focused on the accuracy and intelligibility of educational interpreters who use Cued Speech (CS); this study examines the accuracy of those who use Signing Exact English (SEE). Accuracy, or the proportion of the message correctly produced by the interpreter, was evaluated in 12 SEE transliterators with varying degrees of experience at three different speaking rates (slow, normal, and fast). Results were similar to those previously reported for CS transliterators: (a) speaking rate had a large negative effect on accuracy, primarily due to increased frequency of omissions, (b) the effect of lag time on accuracy was also negative, but relatively small, accounting for just 8% of the variance, and (c) highly experienced transliterators were somewhat more accurate than transliterators with minimal experience, although experience alone did not guarantee accuracy. Lastly, like their CS counterparts, the overall accuracy of the 12 SEE transliterators, 42% on average, was low enough to raise serious concerns about the quality of transliteration services that (at least some) children receive in educational settings.

This series of papers is concerned with the level of access provided by educational interpreters to students with hearing loss. Overall, the approach is to examine the clarity of “visual signals”¹ produced by educational interpreters in as many communication modes as possible. This approach stems from the idea that for students who use interpreters, a clear visual signal is a necessary prerequisite for message understanding, in much the same way as a clear auditory signal (i.e., speech) is necessary for hearing students. The long-term goals of the work are to identify the factors responsible for clarity of the visual signal in each communication mode, by examining two channels in the communication pathway: (a) accuracy, or the percentage of the message correctly produced by the interpreter, and (b) intelligibility, or the percentage of the message that can be correctly received by deaf persons who are proficient in the language and communication mode used by the interpreter. The first two papers in the series examined factors affecting the accuracy (Krause & Tessler, 2016) and intelligibility (Krause & Lopez, 2017) of educational interpreters in the United States who use a communication mode known as Cued Speech (CS; Cornett, 1967). In this article, we examine accuracy in interpreters who use Signing Exact English (SEE).

Signing Exact English

Developed in the early 1970s, Signing Exact English (Gustason, Pfetzing, & Zawolkow, 1972) is a sign system aimed at representing English vocabulary and syntax as literally as possible by providing visual access to English morphology. It uses American Sign Language (ASL) signs, as well as invented signs, in combination with signed representations of English affixes; the invented signs are necessary for (a) representing English grammatical words that do not exist in ASL (e.g., “the”) and (b) differentiating English synonyms that correspond to the same ASL sign. Signs are produced sequentially, in English word order, in conjunction with the mouth movements of English, with the goal of establishing a one-to-one mapping between signs and English words. As a result, the signed representations of English words relating to the same concept such as electric, electrical, electrician, electricity, and nonelectrical are visually distinct (Nielsen, Luetke, & Stryker, 2011). Consequently, it is possible for experienced users of SEE to produce unambiguous representations of syntactic and semantic content in everyday English sentences with a high degree of accuracy (Luetke-Stahlman, 1988).

While the exact number of transliterators² using SEE nationwide is unknown, a survey by Jones, Clark, and Soltz (1997) found that 32.7% of educational interpreting professionals employed in Kansas, Missouri, and Nebraska reported using SEE. In that survey, the only communication option that interpreting professionals used more often than SEE was Pidgin Signed English (PSE) or Conceptually Accurate Signed English (CASE; Winston, 1989). More recent data from the Gallaudet Research Institute’s annual survey (2011) as well as the Educational Interpreter Performance Assessment (EIPA; Schick, Williams, & Kupermintz, 2006) confirm that interpreters who use Manually Coded English systems such as SEE on the job are relatively common, with one EIPA study reporting that those who use SEE, specifically, make up the majority of educational interpreters in some parts of the country (Schick, Williams, & Bolster, 1999).

SEE is an attractive candidate for investigating interpreter accuracy not only because of its sizeable presence in the schools but also because of the direct mapping from the spoken message to the transliterated message; that is, each English word can be represented with signs in only one way. Therefore, it is relatively straightforward to identify and tabulate the number of errors in a transliterated message, resulting in quantitative data that can be used to describe transliterator accuracy in various conditions with a high degree of resolution. This high level of resolution makes it possible to evaluate, quantitatively, the extent to which various factors, such as speaking rate and lag time, affect accuracy of SEE transliterators, which is the purpose of this paper.

Accuracy

While the effect of factors such as speaking rate and lag time on the accuracy of SEE transliterators is relatively unexplored, accuracy itself has been examined more generally for professionals and parents who use SEE. For example, Luetke-Stahlman (1988) reported average sentence-level accuracy ranging from 53% to 89% for seven teachers of the deaf who used SEE. In addition, a study of interpreters and teachers who used one of three English-based sign systems found that while SEE signers’ word-level accuracy was 58% on average, they preserved 86% of the meaning of the message (Luetke-Stahlman, 1991). Finally, it is worth noting that accuracy may improve with intervention; in a small study, five parents improved their signing accuracy substantially after receiving constructive feedback and retained a 10 percentage point improvement in accuracy 3 months after the intervention was completed (Luetke-Stahlman & Moeller, 1990).

More recently, studies using the Educational Interpreter Performance Assessment, or EIPA (Schick et al., 1999) have reported data for more than 2,000 educational interpreters nationwide (e.g., Schick et al., 2006), including those who use Manually Coded English (MCE) systems such as SEE, as well as other communication options such as American Sign Language (ASL), PSE, and Cued Speech (EIPA-CS; Krause, Kegl, & Schick, 2008). These studies have provided highly valuable descriptive information regarding accuracy and other aspects of interpreter performance for interpreters in general and for MCE interpreters, specifically. However, because the EIPA is a holistic evaluation that relies on Likert scale ratings to evaluate a large number of skills, it is not the most convenient tool for studying quantitative effects of various factors on accuracy. Instead, it is more straightforward to use an accuracy-focused metric with more resolution so that small effects can be detected (Krause & Tessler, 2016). Therefore, as in our previous studies (Krause & Tessler, 2016; Krause & Lopez, 2017), a percent-correct measure of accuracy is used in this study so that the effects of both speaking rate and lag time on SEE transliterator accuracy can be evaluated.

Speaking Rate

Understanding the role of speaking rate in transliterator accuracy is essential for characterizing the level of communication access afforded to deaf individuals by interpreting professionals. Yet to our knowledge, this problem has not been widely studied. Instead, what is known comes from a few studies in the visual modality that have examined the effect of speaking rate in situations involving direct communication. These studies have shown a substantial decrease in intelligibility with rate; specifically, reception of artificially accelerated fingerspelling decreases systematically for compression factors greater than two (Reed, Delhorne, Durlach, & Fischer, 1990), intelligibility of isolated ASL signs decreases as compression rate increases, and a breakdown in ASL processing occurs at 2.5–3 times the normal signing rate (Fischer, Delhorne, & Reed, 1999). Ostensibly, it may seem reasonable to expect similar breakdown points for individuals receiving information via an ASL interpreter or SEE transliterator. However, this expectation is based on the visual signal retaining 100% accuracy as speaking rate increases, since the rate alterations in these studies were typically achieved through artificial means (i.e., speeding up video that was signed with 100% accuracy) and resulted in presenting individuals with a compressed, but 100% accurate visual signal for processing.

At present, it is unknown whether ASL interpreters or SEE transliterators could achieve and maintain such a high level of accuracy across a wide range of speaking rates. Our related work on Cued Speech transliterators suggests that instead, it may be more likely for accuracy to decline with increases in speaking rate. In the first paper of this series, Krause & Tessler (2016) reported that the average accuracy of 12 CS transliterators dropped 20 percentage points as speaking rate increased from slow-conversational to fast-conversational rates, with the decline in accuracy caused primarily by an increase in omitted cues. It cannot be assumed, however, that a similar relationship holds for all types of interpreters and transliterators. Therefore, research is needed to determine the nature of the relationship between speaking rate and accuracy for Signing Exact English transliterators.

Lag Time

Another factor that may affect SEE transliterator accuracy is lag time, or the average delay in seconds between the spoken message and the transliterated message. For example, at least one study of ASL interpreters has found a positive association between longer lag times and increased accuracy. In that study, Cokely (1986) analyzed the videotaped performances of four interpreters at a national conference, and reported lower error rates for the two interpreters with longer lag times (i.e., 4 s) than those with shorter lag times (i.e., 2 s). Cokely suggested that the ability to chunk the information into meaningful units resulted in interpreters relaying the most accurate underlying meaning. In addition, he noted that lag time is likely to be proportional to structural differences between the source and target languages. Consistent with this hypothesis, average lag times for CS transliterators (1.86 s; Krause & Tessler, 2016), who work between two forms of the same language (English), are typically shorter than those for ASL interpreters (3 s; Cokely, 1986), who work between two different languages (English and ASL). Perhaps of greater note, however, is that the CS transliterators exhibited an inverse relationship between accuracy and lag time; although the relationship was quite weak and accounted for just 3% of the variance in accuracy, longer lag times were associated with decreased accuracy in CS transliterators. Whether an inverse relationship between lag time and accuracy applies to other types of transliterators such as those who use SEE, however, is unknown. To answer this question, the effect of lag time on accuracy must be examined for SEE transliterators.

Present Study

In the present study, 12 Signing Exact English transliterators of varying experience levels were asked to transliterate an audio source presented at three different speaking rates. Two characteristics of the visual signal were obtained from transliterated messages: (a) accuracy, measured as a percent-correct score based on the target cue sequence, and (b) lag time (in seconds), reported as the average delay between the spoken message and the transliterated message. The goal of the experiment was to determine the effect of speaking rate and lag time on accuracy levels.

Method

Participants

Twelve Signing Exact English transliterators (SEE-T01–SEE-T12) participated in the study. All were female and worked as SEE transliterators (at least part-time) at the time of the study. Each participant completed a written survey regarding experience (in years) as a SEE transliterator, level of education, and relevant certifications. Based on the information provided in this survey, the transliterators were classified into one of three experience categories, following the same definitions that Krause & Tessler (2016) used to classify the experience levels of Cued Speech transliterators:

• novice: minimal certification or no certification, with work experience of less than the equivalent of one full-time year

• experienced: minimal (e.g., state-level) certification with less than the equivalent of three full-time years of work experience, or no certification with 3–5 years of experience

• veteran: highest level of certification (e.g., national-level) and/or more than 5 years of experience.

Table 1 summarizes general background information and the corresponding experience-level classification for each of the twelve participants.

Table 1

Participant background in Signing Exact English transliteration and experience-level classification

	Experience (years)	Current use (hours/week)	Education	Certification	Classification
SEE-T01	2	40	Some college	None	Novice
SEE-T02	12	30–35	Some college	State-level	Veteran
SEE-T03	11	30–35	High school	State-level	Veteran
SEE-T04	8.5	40	Some collegea	State-level	Veteran
SEE-T05	5	37.5	College grada	State-level	Veteran
SEE-T06	8	35	College grada	State-level	Veteran
SEE-T07	12	5–10	Some collegea	State-level	Veteran
SEE-T08	7	6	College grad	State-level	Experienced
SEE-T09	5	1	Some collegea	State-level	Novice
SEE-T10	10	4–5	Some collegea	State-level	Experienced
SEE-T11	7	36	College grad	State-level	Veteran
SEE-T12	6	35	Some college	State-level	Veteran

^aCollege coursework included one or more courses in the field of interpreting or transliterating.

Materials

The stimulus materials used in this study were the same as those used by Krause & Tessler (2016) in an earlier study on the accuracy of Cued Speech transliterators. For that study, an audio lecture about plants was drawn from a 25-min educational film entitled Life Cycle of Plants (Films for the Humanities, 1989). The film, which was designed for use in a high school setting, presents a general introduction to plant growth and reproduction and includes some specialized vocabulary pertaining to plants (e.g., names of plant species). In order to generate a lecture version of this material, the audio narration from the film was re-recorded by a male talker who read a transcript of the film at a deliberate speaking rate; the talker simulated a lecture-style delivery and inserted sizeable pauses (1–3 s) at every sentence (or major phrase) boundary. This recorded audio lecture was then divided into three segments of roughly equal length, not including a short beginning segment (~1 min) of the lecture that was reserved for use as warm-up material. Two additional versions of the lecture materials at alternate speaking rates were also produced using a feature of the computer program Praat (Boersma & Weenink, 2012) that manipulates duration of speech signals without affecting pitch or phonetic qualities. For one version, the lecture was slowed by an expansion factor of 1.25 to obtain a “slow-conversational” speaking rate; for the other version, the lecture was sped up by a compression factor of .8 to obtain a “fast-conversational” speaking rate. Consequently, all lecture materials (warm-up segment and three test segments) were available for the experiment at three different speaking rates: a “normal-conversational” rate (i.e., the original speaking rate) of 109 words-per-minute (wpm), a slow-conversational rate of 88 wpm, and a fast-conversational rate of 137 wpm.

Recording Sessions

Procedures for the recording sessions were identical to those used previously for a study on the accuracy of Cued Speech transliterators (Krause & Tessler, 2016). Specifically, each transliterator participated individually in a single test session that lasted roughly 1.5 h. During the test session, a digital video camera was positioned six to eight feet in front of the transliterator. Each transliterator was given the option to sit or stand, and the camera was centered and zoomed such that a view of the transliterator from the chest and up was achieved. The lecture (warm-up and test) materials were presented from a portable stereo via high-quality speakers, and each participant’s transliteration was filmed and transferred to a computer hard disk for later analysis.

At the beginning of the session, each transliterator was allowed up to 30 min to prepare using a transcript of the lecture and a list of key vocabulary words that was provided by the experimenter. When the transliterator was ready (or when the 30-min preparation time had elapsed), the lecture was presented to the transliterator in three segments; each segment was presented at a different speaking rate. As shown in Table 2, the order of segments was counterbalanced across transliterators to minimize any effects due to differences in difficulty of the segments. The three lecture segments were always presented from the slowest to the fastest speaking rate, such that the first segment presented was at the slow-conversational rate, the second segment at the normal-conversational rate, and the final segment at the fast-conversational rate. Breaks were offered between each of the lecture segments.

Table 2

Presentation order of lecture segments for each transliterator

	Presentation order (rate)
Transliterator	1st (slow-conversational)	2nd (normal-conversational)	3rd (fast-conversational)
SEE-T01, SEE-T07	Segment 1	Segment 2	Segment 3
SEE-T02, SEE-T08	Segment 1	Segment 3	Segment 2
SEE-T03, SEE-T09	Segment 2	Segment 1	Segment 3
SEE-T04, SEE-T10	Segment 2	Segment 3	Segment 1
SEE-T05, SEE-T11	Segment 3	Segment 1	Segment 2
SEE-T06, SEE-T12	Segment 3	Segment 2	Segment 1

Before each lecture segment was presented, a 5-min warm-up period was provided to the transliterator. During this period, the warm-up material was presented at the upcoming speaking rate, which gave the transliterator an opportunity to practice transliterating at the upcoming rate as well as to adjust the volume of the speakers as needed to ensure a comfortable, audible presentation level.

Video Analysis Procedures

Using Adobe Premiere Pro 1.5, the video recordings of each transliteration were viewed in slow motion and transcribed by a highly proficient SEE interpreter with over 10 years of experience in evaluating SEE interpreters for the Board for the Evaluation of Interpreters certification program in the state of Texas. The transcribed signs were then compared to a sequence of target signs (i.e., the sign sequence that would be expected, based on a written transcription of the narrator’s speech). Based on this comparison, each sign was categorized as a correct production, an omission (a target sign that was not produced/attempted by the transliterator), or one of the error types listed below.

• Misproduction error—Transliterator attempted the correct sign or fingerspelling but did not produce it correctly.

• Substitution error—Transliterator produced an incorrect sign.

• Paraphrase error—Transliterator deviated from the original message, using either a different word order, different number of words, or a multiword substitution.

• Insertion error—Transliterator produced an extra sign that was not part of the original message.

The number of signs in each category was tabulated, and percentage scores for each category were calculated by taking the number of signs in the category and dividing it by the total number of target signs for the source message. With this method, the sum of correct signs, omissions, misproductions, substitutions, and paraphrase errors is 100% (i.e., everything spoken in the source message is either signed correctly, signed incorrectly, or omitted). Insertions then push the total beyond 100% in order to reflect how much extraneous signing (beyond what was expected) was introduced to the source message.

Lag time was recorded for each phrase in the source message, where phrases were defined as any utterance separated by the sizeable pauses inserted by the narrator, as described above. Phrase-level lag time was defined as the average of the difference in time between the beginning of the spoken word and the beginning of its visual representation in the transliterated message, measured at three points in time: the first syllable in the phrase, the middle syllable in the phrase, and the final syllable in the phrase.

Results

Table 3 summarizes the overall frequency of occurrence for each of the sign production categories, averaged across all transliterators and speaking rates. As expected, correct productions occurred more frequently than any other category; however, the average proportion of correct signs was just 42%. Omissions were the most frequent type of error (40%), followed by substitutions (10%), misproductions (6%), and attempts to paraphrase (2%). Insertions occurred relatively infrequently, resulting in an extra 2% of signs beyond what was expected from the target sign sequence.

Table 3

Average frequency of occurrence for sign production categories and average lag time by individual transliterator

	Sign production category
	Correct signs (%)	Production errors (%)	Omissions (%)	Substitutions (%)	Paraphrase (%)	Insertions (%)	Lag time (sec)
SEE-T01	37.8	9.0	44.7	7.9	.6	3.1	4.30
SEE-T02	45.5	6.9	38.3	7.2	2.0	3.1	3.94
SEE-T03	46.0	5.4	37.8	7.2	3.6	2.6	4.10
SEE-T04	38.1	4.0	47.1	10.2	.6	1.9	3.04
SEE-T05	68.0	4.4	21.6	5.0	1.0	1.4	2.54
SEE-T06	26.2	8.5	47.6	16.7	.9	2.3	3.33
SEE-T07	42.8	6.3	37.7	11.8	1.5	1.8	2.56
SEE-T08	40.9	4.0	42.2	10.6	2.3	1.7	3.65
SEE-T09	41.7	7.1	41.8	7.9	1.5	.7	3.25
SEE-T10	25.0	4.9	53.4	12.9	3.8	2.5	3.12
SEE-T11	35.1	7.4	39.5	11.3	6.8	3.0	3.91
SEE-T12	53.2	6.4	30.3	8.6	1.5	1.7	2.56
Average	41.7	6.2	40.2	9.8	2.2	2.2	3.36
SE	3.3	.5	2.4	.9	.5	.2	.17

A repeated-measures, one-way analysis of variance performed on the frequency of occurrence data (after an arcsine transformation to reduce any inequalities in variance) showed that the main effect of sign production category was significant [F(1.52,16.71) = 138, p < .001; η_p² = .926]. Post hoc tests with Bonferroni corrections to adjust for multiple comparisons found no significant difference between frequency of correct signs and frequency of omissions (p = 1.0), which both occurred much more often than all other error categories (p < .01; 2.34 ≤ d ≤ 6.45). Substitutions occurred more frequently than misproductions (p = .033; d = 1.15), which were in turn more frequent than both paraphrase attempts and insertions (p < .01; 1.66 ≤ d ≤ 3.14). There was no statistically significant difference in the frequency of paraphrase attempts and insertions (p = 1.0).

Data for individual transliterators, also shown in Figure 1, revealed some differences in frequency of production patterns among participants. Although average frequencies were similar for correct sign productions and omissions, this pattern was not the case for every individual. Rather, frequency of correct productions ranged from 25% (SEE-T10) to 68% (SEE-T05) for individuals, while the proportion of the message that was omitted also varied considerably from 22% (SEE-T05) to 53% (SEE-T10). Given that the frequency of the other error types did not differ dramatically across transliterators, the result was a general trend for accuracy (i.e., frequency of correct signs) to increase as frequency of omissions decreased. Omissions were the most frequent type of error for every transliterator, and of the seven transliterators with the lowest accuracy, six omitted more than 40% of the target signs in the source message (SEE-T01: 45%, SEE-T09: 42%, SEE-T10: 53%, SEE-T08: 42%, SEE-T06: 48%, and SEE-T04: 47%). Notably, this group included all four of the transliterators at the lower two experience levels. Consequently, the average accuracy for transliterators at the highest experience level (i.e., veterans) was higher on average than those with less experience (44% vs. 36%); this differential was primarily a result of fewer omissions—veterans omitted 37% of signs on average, while novice and experienced transliterators omitted 45% and 48%, respectively. Although this difference could have occurred by chance, the size of the difference between the two groups raises the possibility that experience may also play a role in accuracy. Regardless, it must be noted that experience alone is not a guarantee that adequate transliteration skills are developed. For example, two of the eight transliterators classified at the veteran experience level, SEE-T04 and SEE-T06, also exhibited a large proportion of omissions (leaving out more than 40% of the target sign sequence), and SEE-T06 had more frequent substitutions (17%) and lower accuracy (26%) than any other transliterator in the study.

Figure 1

Frequency of occurrence for each sign production category (correct signs, misproductions, omissions, substitutions, paraphrase attempts, insertions) for the individual transliterators at each of the three experience levels.

Effect of Speaking Rate

The effect of speaking rate on each of the sign production categories is shown in Figure 2. Six repeated-measures, one-way analyses of variance (one for each sign production category, with speaking rate as a within-subjects factor) indicated that the frequency of omissions [F(2, 22) = 101.2, p < .001; η_p² = .902], correct signs [F(2, 22) = 41.0, p < .001; η_p² = .788], misproductions [F(2, 22) = 14.8, p < .001; η_p² = .573], and insertions [F(2, 22) = 8.1, p = .002; η_p² = .427] differed significantly across speaking rates. Post hoc tests with Bonferroni corrections to adjust for multiple comparisons showed that the proportion of omissions increased significantly with each increase in rate (p < .001; 1.68 ≤ d ≤ 4.07), while the proportion of correct signs decreased significantly with each increase in rate (p < .01; 1.11 ≤ d ≤ 2.30); that is, omissions exhibited a strong positive (i.e., direct) relationship with speaking rate, while correct signs showed a strong negative (i.e., inverse) relationship. Misproductions also had a negative relationship with rate, although not quite as strong; significant decreases in the frequency of misproductions were detected (p < .05; .74 ≤ d ≤ 1.60) at each change in speaking rate. A weaker negative relationship with rate was evident for insertions, with frequency of insertions at the slow rate significantly higher than at normal and fast rates (p < .05; d = 1.02 and d = .90, respectively) but no statistically significant difference between normal and fast rates (p = 1.0). While the proportion of substitutions suggested a weak negative relationship with speaking rate (decreasing from 10.6% at the slow rate to 9.3% at the fast rate), this overall difference in frequency of substitutions across speaking rate did not reach statistical significance [F(2,22) = 2.2, p = .135]. Paraphrase attempts were generally unaffected by rate [F(2,22) = .4, p = .683].

Figure 2

Frequency of occurrence for each sign production category (correct signs, omissions, substitutions, misproductions, paraphrase attempts, insertions) at each of the three speaking rates, averaged across all transliterators.

Data for individual transliterators was consistent with these patterns. Figure 3 illustrates the very large effect of rate on both correct signs and omissions, the sign production categories that occurred by far the most frequently. Panel B shows that every transliterator’s frequency of omissions increased with each increase in speaking rate, although the change for SEE-T05 from slow to normal rates was fairly small, only about 3 percentage points. Similarly, the frequency of correct signs decreased for all individual transliterators as speaking rate increased from slow to normal rates (Panel A)—again, SEE-T05 showed the smallest change (again, about 3 percentage points). The decline in frequency of correct signs continued from normal to fast rates and was fairly large for all but two transliterators, SEE-T04 and SEE-T10, who both showed little difference in accuracy between normal and fast rates. Figure 4 shows the role of rate in the sign categories that were less frequently occurring. Most transliterators also showed small declines with speaking rate in misproductions (all but SEE-T05—see Panel B), at least between slow and fast rates, as well as insertions (Panel D), consistent with a somewhat smaller effect of rate. Although several transliterators also showed some decrease in substitutions (Panel A) between the slow and fast speaking rates, the amount of decrease was less consistent, ranging from one percentage point (SEE-T01 and SEE-T06) to 6 percentage points (SEE-T04). Finally, no transliterator showed substantial changes in frequency of paraphrase with speaking rate (Figure 4, Panel C). Thus, the effect of speaking rate on sign production was very robust. As speaking rate increased from slow to fast rates, every transliterator’s accuracy decreased, and the decrease was primarily a result of increased omissions.

Figure 3

Individual data for the two sign production categories that occurred most frequently (correct signs, omissions). Each graph shows frequency of occurrence at the three speaking rates, for each of the 12 transliterators. Experience categories of individual transliterators are denoted by line style: dotted lines for novices, dashed line for experienced, and solid lines for veterans.

Figure 4

Frequency of occurrence for the four sign production categories that occurred less frequently (misproductions, substitutions, paraphrase attempts, insertions). Each graph shows frequency of occurrence at the three speaking rates, for each of the 12 transliterators. Experience categories of individual transliterators are denoted by line style: dotted lines for novices, dashed line for experienced, and solid lines for veterans.

Effect of Lag Time

Although some transliterators used varying amounts of average lag time at each of the three different speaking rates, a repeated-measures, one-way analyses of variance found no statistically significant effect (p > .05) of speaking rate on average lag time. Therefore, the effect of lag time on accuracy was examined using the overall average lag time (averaged across all three speaking rates) of the twelve individual transliterators, shown in Table 3. These data show that average lag time varied considerably across transliterators, ranging from 2.54 s (SEE-T05) to 4.30 s (SEE-T01). Yet despite this wide variation, no relationship (p > .05) could be detected between a transliterator’s average lag time and average accuracy (i.e., the average frequency of correct signs produced by the transliterator across all three speaking rates).

At the phrase level, however, a weak negative relationship (Pearson’s r = −.281, p < .001) between lag time and accuracy was detected at the slow speaking rate (for which phrase-level accuracy was available). Figure 5 shows this relationship, depicting the lag time and accuracy of each phrase produced at the slow speaking rate by all 12 transliterators. Transliterators used lag times ranging from .66 s to 10.37 s when signing individual phrases spoken at the slow rate, but most phrases (75%) were produced with lag times between 1.25 s and 4.75 s, and the vast majority (95%) of phrases were produced with lag times of 6.5 s or less. As Figure 5 shows, however, any particular lag time was associated with a wide range of accuracies (from 20% or less to 100%). Thus, the tendency for accuracy to decrease as lag time at the phrase level increased was fairly weak; although the relationship between these two variables was statistically significant, it accounted for just 8% of the variance in phrase-level accuracy.

Figure 5

Relationship between accuracy, or percentage of cues correctly produced, and lag time at the phrase-level. Each data point represents one phrase produced by one transliterator. The best-fitting line indicates a weak, negative relationship between the two variables.

Supplemental Analyses

In addition to the analyses and results described above, supplemental analyses were conducted to further explore transliterator accuracy in three areas. The first set of measurements investigated the role of mouthing, or the expectation that a transliterator silently produce the English source message on the mouth while simultaneously signing the message in SEE. The second set of measurements examined transliterators’ use of paraphrase in more detail. The final set of measurements examined the nature of substitution and misproduction errors. These measurements and the corresponding results are described in more detail below.

Role of Mouthing

The expectation that SEE transliteration includes mouthing of the English source message means that, even when the sign produced by the transliterator is incorrect, it may nevertheless be accompanied by accurate mouth movements. In order to explore this possibility, the expected mouth movements for each English word in the source message were categorized in a manner analogous to that used for the sign categories. Specifically, mouthing for each English word in the source message was categorized as either a correct production (mouthed source word appropriately), a substitution (mouthed something other than the source word, even though the sign was correct), a paraphrase (mouthed words in a different order or a multiword substitution), an omission (produced no mouth movements at all), or an insertion (for any words that were mouthed but not signed). As Table 4 shows, more than 98% of all mouth movements were either correct or omitted, so only those two categories will be discussed here. The relationship of these two categories was similar to the signed message; correct mouth movements occurred most frequently, declining in frequency as speaking rate increased, and this decline was explained by an increase in omissions. However, the proportion of correct mouth movements was at least 10 percentage points higher than the proportion of correct signs, for every transliterator at every speaking rate, and was more than 20 points higher on average (64% correctly mouthed vs. 42% correctly signed). Moreover, the proportion of omitted mouth movements was slightly lower than the proportion of omitted signs for nearly every transliterator and speaking rate and was 6 points lower on average (34% omitted mouth movements vs. 40% omitted signs). Thus, transliterators maintained correct mouth movements for most signs that were produced incorrectly (i.e., substitutions or misproductions) as well as some signs that were omitted altogether.

Table 4

Average percentage of original message words mouthed correctly and omitted on the mouth by each transliterator by speaking rate

	Slow rate		Normal rate		Fast rate		Average
	Correct mouth (%)	Omitted mouth (%)	Correct mouth (%)	Omitted mouth (%)	Correct mouth (%)	Omitted mouth (%)	Correct mouth (%)	Omitted mouth (%)
SEE-T01	75.7	23.9	55.6	43.3	47.0	52.8	59.4	40.0
SEE-T02	80.5	19.5	61.9	37.4	51.9	47.5	64.7	34.8
SEE-T03	69.9	29.2	64.3	35.1	46.5	52.9	60.2	39.1
SEE-T04	71.5	27.3	60.1	38. 9	52.0	45.8	61.2	37.3
SEE-T05	92.4	7.0	87.6	12.0	69.4	30.1	83.2	16.4
SEE-T06	68.9	29.6	53.7	44.4	49.7	45.9	57.4	40.0
SEE-T07	86.1	13.5	73.0	25.1	58.0	37.8	72.4	25.5
SEE-T08	77.6	20.4	56.0	41.5	47.38	50.3	60.3	37.4
SEE-T09	76.9	21.2	69.5	28.7	51.0	47.3	65.8	32.4
SEE-T10	59.2	36.3	42.5	54.2	41.9	54.2	47.9	48.2
SEE-T11	72.3	26.5	57.3	41.4	46.5	52.3	58.7	40.1
SEE-T12	90.1	9.7	81.1	18.5	62.8	34.9	78.0	21.1
Average	76.8	22.0	63.6	35.0	52.0	46.0	64.1	34.3
SE	2.7	2.5	3.6	3.5	2.3	2.2	2.8	2.6

Use of Paraphrase

Paraphrase, while not an accurate verbatim transliteration of the message, can still be an effective way of preserving message content. Each paraphrase attempt used by transliterators in this study was reviewed and classified according to whether or not it (a) preserved the meaning of the original message, and (b) also used acceptable English grammar. This review revealed that just over half (53%) of paraphrase attempts used by transliterators in this study preserved the meaning of the original message, with a substantially lower percentage (29%) also maintaining acceptable English grammar. Notably, individual transliterators varied considerably in proportion of paraphrase attempts that were effective in preserving the meaning of the original message, from 0% for SEE-T06 to 100% for SEE-T05.

Nature of Substitution and Misproduction Errors

To better understand the substitution and misproduction errors in this study, each error type was further classified according to the nature of the error. Specifically, all misproduction errors were further classified into one of four types of misproductions; the breakdown was as follows: (a) phonetic errors—64% (e.g., error in movement, handshape, or location), (b) minor fingerspelling production problems—24%, (c) overinitialization, or initializing a basic sign that is uninitialized in SEE—2% (e.g., using the L handshape to sign LONG), and (d) failure to fingerspell a word that has no SEE sign—10% (i.e., choosing an acceptable ASL or invented sign but not pairing it with fingerspelling as required by SEE, or combining a SEE sign with partial spelling of the word instead of spelling the whole word—such as signing WORK and spelling #AHOLIC for the word “workaholic”). Similarly, substitution errors were placed into one of seven categories, according to the nature of the error. These categories closely follow the classification scheme used by Cokely (1986) in a study of the relationship between accuracy and lag time in ASL interpreters; two additional categories were necessary to cover substitution types specific to SEE transliteration. A description of each of the seven categories is below, along with the proportion of substitution errors in the study that fell into each category:

• Traditional: 30%—matched concept, but used traditional ASL sign rather than SEE sign

• General: 17%—meaning preserved but not a verbatim transliteration

• Expansive: 1.4%—added meaning unintended in original message (see also Cokely, 1986)

• Restrictive: 1.7%—omitted partial meaning from original message (see also Cokely, 1986)

• Cohesive: 3.4%—meaning preserved but grammar is altered or incorrect (see also Cokely, 1986)

• Unrelated: 33%—meaning was not consistent with original message (see also Cokely, 1986)

• Anomalous: 13%—contained multiple errors or was otherwise unintelligible (similar to the general Anomalies category as defined by Cokely, 1986)

Discussion

The primary purpose of this study was to examine the effects of two factors, speaking rate and lag time, on the accuracy of messages produced by Signing Exact English transliterators. Results for both factors were very similar to those previously reported for Cued Speech transliterators (Krause & Tessler, 2016). First, although both factors played a statistically significant role in overall accuracy, the effect of lag time on accuracy was quite small. There was no relationship between a transliterator’s average lag time and average accuracy, and the relationship between these variables at the phrase level (decreased accuracy with increased lag time) was relatively weak, accounting for only 8% of the variance in phrase-level accuracy. Second, speaking rate had a large negative effect on accuracy. On average, SEE transliterators were 20 percentage points less accurate at the fast speaking rate than at the slow speaking rate. This decline was caused primarily by an increased proportion of omitted signs (22 percentage points more, on average) over the same range of speaking rates. This effect was robust, with every individual transliterator following the pattern.

Accuracy

This study was not designed to characterize the average accuracy of typical SEE transliterators; rather, as the third paper in a series concerned with the level of access afforded to students who use educational interpreters, the purpose was to determine if and how factors such as speaking rate and lag time are related to accuracy. However, it must be said that the overall accuracy, 42% on average, is sobering and raises concerns—similar to those raised previously for CS transliterators (Krause & Tessler, 2016)—regarding the quality of transliteration services that (at least some) children receive in educational settings. While the 12 educational SEE transliterators in this study do not constitute a large or nationally comprehensive sample, they were all employed as classroom interpreters at the time of the study and came from multiple school districts in two separate states. Thus, they likely represent at least some segment of “typical” transliterators working in the schools and their data suggest an urgent need for increased transliterator training and professional development opportunities. Unfortunately, however, such opportunities are limited. Of the interpreter training programs in the country, none provide training specifically for SEE transliteration. Furthermore, most transliterators in this study reported that opportunities for professional development of their SEE skills typically consisted of just one weeklong workshop per year.

While it is likely that more frequent training opportunities would improve overall accuracy, it is also worth noting that even though transliterator accuracy in this study was just 42%, this number is only an average, and higher accuracy was observed in some circumstances. At the slow speaking rate, for example, the accuracy was 52% on average, with all individual transliterators exhibiting higher accuracy at the slow rate than at the normal or fast rates. Moreover, this finding suggests that presenting the message at an even slower rate than the one used in this study may lead to further increases in accuracy. Of course, a speaking rate slower than the slow rate used in this study is not likely to be a typical rate for conversational communication. However, as we have stated previously (Krause & Tessler, 2016), (a) it is possible that it may nonetheless be an appropriate representation of a deliberate pace used by some classroom teachers when presenting complex material, and, therefore, (b) it may be helpful for future studies to measure speaking rates of teachers in actual classrooms in order to determine what range of speaking rates is most appropriate to evaluate.

In addition to speaking rate, another factor that has a possible role in accuracy is transliterator experience. While no firm conclusions regarding the role of experience can be drawn from the results of this study, there was a general trend for increased experienced level to be associated with increased accuracy. Specifically, the accuracy of veteran transliterators was 44% on average (compared to 36% for those with less experience), and this increased to 54% at the slow speaking rate (compared to 47% for those with less experience). Moreover, the transliterator who had by far the highest accuracy overall was a veteran: SEE-T05’s accuracy was 68% on average and 77% at the slow rate. However, despite these indicators suggesting that accuracy may increase with increased experience, experience alone did not guarantee accuracy; veteran transliterators exhibited a wide range of accuracy levels from 26% for SEE-T06 to 68% for SEE-T05. This high degree of variation in performance levels, even for highly experienced transliterators, underscores the importance of assessing all transliterators individually and quantitatively, regardless of their level of experience.

Intelligibility

Despite the relatively low accuracy of transliterators in this study overall, it is worth noting that their message intelligibility (percentage of original spoken message correctly received by deaf consumers proficient in Signing Exact English) is likely to be somewhat higher. In a recent study of Cued Speech transliterators, for example, overall message intelligibility was more than 10 percentage points higher than average accuracy (72% vs. 61%) of the cued messages produced by 12 transliterators (Krause & Lopez, 2017). Furthermore, supplemental analyses of the data in this study suggest that there are at least three reasons to expect a similar relationship between accuracy and intelligibility for the transliterators in this study. First, accuracy of mouth movements by transliterators in this study was considerably higher on average than accuracy of signs (22 percentage points). This difference is consistent with the expectation that message intelligibility is likely to be somewhat higher than message accuracy, since deaf consumers may be able to use information on the mouth (particularly in cases where sentence context is also available from neighboring correct signs) to identify the target English word, even when it was not signed correctly.

Second, more than half of all paraphrase attempts used by transliterators in this study preserved the meaning of the original message. Even though the paraphrase itself is not a verbatim representation of the message, effective use of paraphrase could lead to increased verbatim intelligibility if the additional context provided by the paraphrase can aid deaf consumers in decoding subsequent inaccuracies in the signed message. That said, the number of paraphrase attempts, overall, by transliterators in this study was relatively low, presumably because they were instructed to provide a verbatim transliteration of the English message. As Table 3 shows, the amount of paraphrase used by transliterators ranged from .6% of the message by SEE-T01 and SEE-T04 to 6.8% of the message by SEE-T11 and was just 2.2% on average across all transliterators. Taken together with large individual differences in transliterators’ effectiveness in preserving the meaning of the original message (0–100%), these findings suggest that the use of paraphrase may be helpful to message intelligibility (relative to message accuracy) for some transliterators, but the effect is likely to be fairly small.

One final factor that could affect message intelligibility is the nature of errors in the signed message. It is possible that some types of errors may be more detrimental to intelligibility of the message than other types of errors. For example, omissions from the signed message, the most common type of error in this study in this study by far, are likely among the more catastrophic types of errors, because deaf consumers are left with no information regarding the target word in the original message (at least when corresponding mouth movements are also omitted) other than context. On the other hand, other types of errors such as substitutions and misproductions may sometimes be relatively close to the target word in the original message and thus be relatively easy for deaf consumers to recognize (particularly, when in context and accompanied by appropriate mouth movements).

An inspection of the categories of misproduction errors (phonetic errors, minor fingerspelling production probems, overinitialization, and failure to fingerspell a word that has no SEE sign) suggests that each one provides at least some correct information about the underlying English target word in at least some instances. Therefore, it seems likely that at least some of the misproductions in this study, which occurred on 6.2% of target signs (Table 3), may still be intelligible to deaf consumers despite the misproduction error. Furthermore, this possibility seems most likely in cases where the misproduced sign is accompanied by appropriate English mouth movements and surrounding context is available. A similar case can be made for some, but not all, of the substitution categories. Specifically, two categories, Unrelated and Anomalous, do not provide any correct information about the target English word, while four categories, General, Expansive, Restrictive, and Cohesive, do provide at least some correct information; these four categories describe 23.5% of all substitutions that occurred in the study. The final category, Traditional, accounts for 30% of substitutions and provides information only to those deaf consumers of SEE who are also familiar with ASL. Given that substitutions occurred on 9.8% of target signs (Table 3) in this study, deaf consumers of SEE would thus have some correct information about the signs produced with substitution errors for somewhere between 2.3% (if they were unfamiliar with ASL) and 5.2% (if they had ASL knowledge) of the target sign sequence. It is possible, then, that at least some portion of these signs, may still be intelligible to deaf consumers, particularly when accompanied by appropriate English mouth movements and context.

Conclusions

Although more research is needed, the results of the present study provide valuable quantitative information regarding two factors that affect the accuracy of Signing Exact English transliterators. By using methods similar to the first study in this series (Krause & Tessler, 2016) to examine transliterators of varying experience levels and lag times at three different speaking rates, three important findings have been established. First, the effect of lag time on accuracy was relatively small, with longer lag times only weakly associated with lower accuracy. Second, experience does not guarantee accuracy. However, preliminary data in this study suggest that highly experienced transliterators are likely to be somewhat more accurate than transliterators with minimal experience. This possibility should be examined in future research by investigating larger numbers of transliterators at every experience level, from a wide variety of backgrounds (i.e., various credentials, amounts, types of training, etc.), for the relationship between transliterator characteristics (e.g., working memory, manual dexterity, etc.) and transliterator accuracy. Third, faster speaking rates are associated with reduced transliterator accuracy, primarily due to increased frequency of omissions. Although higher numbers of omissions are an understandable coping strategy for transliterators facing increases in speaking rates, the impact of this strategy on a deaf child’s ability to receive the message is unknown. Therefore, an important next step is to investigate SEE transliterator intelligibility as a function of speaking rate and accuracy. While intelligibility is likely to be somewhat higher than accuracy, the exact relationship must be quantified at various speaking rates in order to gain insight into how these factors affect the level of access afforded by SEE transliterators. In addition, it is equally important to extend this work to educational interpreters who use other communication modes with deaf children in classrooms (e.g., ASL, CASE, etc.) using similar accuracy and intelligibility experiments. Such research is critical for determining what accuracy levels are needed from interpreters/transliterators in each modality at each speaking rate in order to ensure accessibility for all children with hearing loss who use interpreters and transliterators in the classroom.

Funding

This work was supported in part by a grant from the National Institute on Deafness and Other Communication Disorders/National Institutes of Health (Grant no. 5 R03 DC 007355).

Conflict of interest

No conflicts of interest were reported.

Acknowledgments

The authors wish to thank Andrew Hague for assistance with lag time measurements, and John Lum for assistance in stimulus creation.

Footnotes

1

The term “visual signal” used here is a shortened version of Battison’s (1978) observation that sign language is “a manually produced, visually received signal” while speech is “an orally produced, auditorily received signal.”

2

Whereas the function of an interpreter is to translate between two languages (e.g., spoken English and ASL), the function of a transliterator is to transfer information between two modes of the same language (e.g., spoken English and either signed English or cued English).