Abstract

This study examined the effectiveness of cognitive training (the Thinking Skills for Work Program) for improving competitive employment outcomes in persons with severe mental illness who have a history of job failure. Forty-four clients receiving services at 2 inner-city community mental health agencies were randomly assigned (within each site) to receive either cognitive training and supported employment (CT + SE) or supported employment only (SE Only). Retention in the CT + SE program was high (91%). Baseline to 3-month assessments showed significantly greater improvements in neurocognitive functioning, depression, and autistic preoccupation on the Positive and Negative Syndrome Scale for the CT + SE set compared to the SE Only group. Employment outcomes over 1 year showed that clients who received CT + SE were significantly more likely to work (69.6 versus 4.8%, respectively), worked more jobs, worked more hours, and earned more wages than clients with SE Only. The findings provide support for the feasibility of implementing the Thinking Skills for Work Program in the context of supported employment and its beneficial effects on cognitive functioning and competitive employment in persons with severe mental illness.

Introduction

Competitive employment rates for persons with severe mental illnesses such as schizophrenia, bipolar disorder, and treatment-refractory depression are low, with most estimates ranging between 10 and 20%.1–3 Despite this low rate of employment, most individuals with severe mental illness express a desire to work.3–4 Furthermore, work has been found to be associated with a range of benefits including improved self-esteem, less severe symptoms, better economic standing, greater satisfaction with finances, and an improved sense of recovery.5–8

Over the past decade, supported employment programs have been developed as a vocational rehabilitation approach in helping people with severe mental illness achieve their employment goals.9 Supported employment is distinguished from other approaches to vocational rehabilitation for persons with severe mental illness by its emphasis on rapid job search (rather than extensive prevocational assessment or training), competitive jobs in integrated community settings (rather than sheltered or set-aside jobs for persons with disabilities), integration of vocational and clinical services (rather than having separate vendors provide them), attention to consumer preferences with respect to job types and disclosure of psychiatric disorder to employers (rather than jobs chosen for clients and mandatory disclosure), and follow-along supports to facilitate job maintenance (rather than ending vocational services when the person has successfully attained a job).10 Empirical support for these programs has been strong, with multiple randomized controlled trials demonstrating the superiority of supported employment over a variety of other rehabilitation models, including group skills training, sheltered workshops, psychosocial rehabilitation programs, and brokered vocational services.11–18 The evidence for supported employment from controlled trials is compelling, it is widely accepted as an evidence-based practice for severe mental illness, and efforts are under way to disseminate it.19

Despite the superiority of supported employment over other models of vocational rehabilitation for persons with severe mental illness, not all individuals benefit from it. Significant numbers of persons in supported employment programs work very little or not at all, with most estimates ranging from 25 to 75% of participants.19 Among clients who work, job tenure is often brief, usually averaging between 3 and 5 months,13–14, 16, 18, 20–22 and unsuccessful job endings, such as being fired or walking off the job without another job in place, are common.23–24 Considering the importance of work for clients with severe mental illness, and the limited benefits many people experience from supported employment programs, there is a need to develop strategies for enhancing the effectiveness of these programs.

One potentially important focus of intervention for improving the outcomes of supported employment is cognitive functioning. There are several reasons why improving cognitive functioning in persons with severe mental illness may increase their ability to benefit from supported employment services. First, cognitive impairment is a common feature in severe mental illnesses,25 especially schizophrenia,26 suggesting it is a problem frequently present in persons receiving supported employment services and a potential target for remediation. Second, cognitive functioning is related to employment outcomes in persons with severe mental illness retrospectively, concurrently, and prospectively and in general psychiatric samples as well as in individuals receiving supported employment or other vocational services.27 Thus, poor cognitive functioning appears to contribute to worse employment outcomes, even in individuals receiving vocational supports. Third, there is a growing literature indicating that cognitive rehabilitation can produce modest improvements in cognitive functioning,28–29 although there continues to be a debate as to whether these improvements translate into functional gains in domains such as social functioning, self-care, and work.30 These findings suggest that systematic efforts at improving cognitive functioning are feasible, although special consideration may be required in order to ensure that these gains are transferred to employment settings. The present article describes the results of a cognitive training intervention designed to improve competitive employment outcomes in persons with severe mental illness who have experienced job failure.

The Thinking Skills for Work Program

The Thinking Skills for Work Program was designed as an adjunct to supported employment and is aimed at integrating cognitive rehabilitation with the ongoing provision of supported employment services. It is divided into 4 component parts and is delivered by a cognitive training specialist who works collaboratively with the supported employment specialist.

Assessment

During the assessment phase a thorough cognitive assessment of the client is conducted, as well as a detailed employment history. In addition to gathering information about the number, types, and duration of past employment experiences, a job loss analysis is conducted on the most recent 1 or 2 jobs, aimed at evaluating the possible contribution of cognitive impairment to the losses. These results are discussed in a meeting with the cognitive training specialist, the supported employment specialist, and the client. This meeting highlights the client's strengths and motivation to work and creates positive expectations for the potential benefits of the Thinking Skills for Work Program for improving the client's cognitive functioning and achieving his or her employment goals.

Computer Cognitive Training

Clients are engaged in approximately 24 hours of computer-based cognitive exercises (Cogpack, version 6.0, Marker Software) providing practice across the broad range of cognitive functions that are impaired in persons with severe mental illness, including attention and concentration, psychomotor speed, learning and memory, and executive functions. Exercises practicing all of these areas of cognitive functioning are included within the first 6 cognitive training sessions, with additional sessions focusing on further practice. Sessions require 45–60 minutes to complete, with clients usually completing 2–3 sessions per week for a total duration of about 12 weeks. Participants receive performance scores reflecting accuracy and speed following the completion of each exercise, which are recorded and referred to in order to reinforce performance progress. Computer exercises are designed to be enjoyable and reinforcing to complete, with difficulty gradually increasing over time.

Job Search Planning

The cognitive training specialist, the employment specialist, and the client meet together to plan the job search, based on the client's vocational preferences. Some clients choose to delay the job search until they have partially or entirely completed the computer cognitive training exercises. Others prefer to commence the job search concurrent with participating in the cognitive training exercises. The client chooses when he or she wants to begin looking for a job. Then, a meeting is held to review the client's job interests, to evaluate his or her cognitive strengths and gains made in the computer cognitive training exercises (when appropriate), and to consider possible needs or supports the client may require to compensate for any persisting cognitive impairments that could compromise work performance.

Job Support Consultation

Following job attainment, the cognitive training specialist and the employment specialist meet regularly, sometimes with and sometimes without the client, to discuss job supports to address cognitive challenges on the job. These supports are designed to enhance the transfer of cognitive skills in the computer training exercises and to minimize the effects of any persisting cognitive impairments.

Broadly speaking, cognitively based employment supports can be divided into those that are remediative (or restitutive) in nature and those that are compensatory in nature. Remediative supports are designed to improve cognitive functioning at the workplace, such as repeatedly practicing a task in order to improve psychomotor speed. Compensatory strategies do not directly effect cognitive processes themselves but, rather, minimize the negative impact of compromised cognitive functioning on work behavior. For example, teaching the client to use memory devices (such as writing items down so they will not be forgotten or asking the supervisor for help when a problem arises) can reduce the effects of memory or executive function impairments at the workplace without directly influencing these cognitive capacities. Some strategies may be both compensatory and remediative in nature.28 For example, scheduling breaks is a strategy for compensating for limited attention span, but by gradually extending the work periods between breaks, one's span of attention may also be increased, thus remediating partially or entirely the difficulties in attention.

Program Evaluation

The Thinking Skills for Work Program was designed to overcome the limitations of much prior research on cognitive rehabilitation in 2 ways. First, reviews of cognitive rehabilitation research suggest that there is limited transfer of cognitive skills trained in laboratory settings to “real-world” locations where cognition has its effect on functional outcomes.29–30 The Thinking Skills for Work Program addresses this problem by combining computer cognitive training exercises with cognitively informed job search planning, as well as the development of remediative and compensatory cognitive strategies to address on-the-job functioning. Integrating the laboratory-based cognitive training exercises with job planning and maintenance strategies that are informed by information about the person's cognitive strengths and weaknesses, and gains in the computer exercises, provides opportunities for both optimizing the transfer of cognitive skills to in vivo work settings and minimizing the effects of any persistent cognitive deficits on work performance.

Second, while some cognitive rehabilitation programs have sought to combine cognitive rehabilitation with other established methods for improving outcomes such as social functioning,31–32 many others have not. In fact, despite the strong association between cognition and vocational functioning in persons with severe mental illness, and the fact that supported employment is the most effective intervention for improving competitive work in this population,19 no studies have been published examining the effects of cognitive rehabilitation on work in persons participating in supported employment.

This article describes the effects of a cognitive training program for clients participating in 2 supported employment programs for severe mental illness. The major hypotheses were that, compared to clients who received supported employment services alone, clients who participated in the Thinking Skills for Work Program (and also received supported employment) would be more likely to obtain competitive work, would work more hours, and would earn more wages.

Method

Forty-four clients with severe mental illness receiving services at 2 community mental health centers in Brooklyn, New York, were randomly assigned (within each center) to 1 of 2 programs: cognitive training and supported employment (CT + SE) or supported employment only (SE Only). Comprehensive employment data were collected for 1 year, and cognitive and psychopathology assessments were conducted at baseline and 3 months later.

Participants

Eligibility criteria for participation in the study included (a) severe mental illness as determined by the State of New York Office of Mental Health; (b) not currently competitively employed; (c) desire for competitive work; (d) currently enrolled in supported employment program; (e) evidence of prior job failure, defined as getting fired from a job held for less than 3 months or walking off a job without another job in place after less than 3 months of employment; and (f) willingness and capability of giving informed consent to participate in the study. Clients were recruited through referrals from supported employment and other rehabilitative and clinical staff at the agencies and self-referrals by clients who had heard about the program from other clients or staff. After completion of the baseline interview, clients were randomly assigned to either the CT + SE program or the SE Only program. Assignment to the programs was conducted on the basis of a computer-generated randomization list. The total of 48 clients was randomized, including 25 in CT + SE and 23 in SE Only. Soon after randomization, 2 clients (1 CT + SE, 1 SE Only) withdrew consent from the study due to serious medical conditions that precluded them from working, and 1 client (SE Only) died. One client left vocational services and was lost to follow-up within a month after randomization to SE Only. Thus, the final randomized sample included 23 clients in CT + SE and 21 clients in SE Only. The demographic and clinical characteristics of the study participants are summarized in table 1.

Study Sites

The study took place at 2 community mental health centers (CMHC A and CMHC B) in Brooklyn, both serving predominantly minority clients, and both providing a comprehensive range of services, including housing, psychiatric, community support, day treatment, psychosocial, and integrated vocational services (including supported employment), to adults, children, and families with severe mental illness, mental retardation, and/or substance abuse.

Supported Employment Programs

Both sites had supported employment programs that broadly adhered to evidence-based definitions of the practice.19 Both programs had zero exclusion criteria, no prevocational training, minimal prevocational assessment, emphasis on rapid job search for competitive employment in integrated community settings, attention to consumer preferences with respect to jobs sought and disclosure of psychiatric disability, and provision of follow-along supports to facilitate job retention.

An assessment of the fidelity of the 2 programs to the principles of supported employment was conducted by McGurk and Mueser with the Supported Employment Fidelity Scale.33 This scale contains 15 items, each rated on a 5-point anchored Likert scale, with items tapping 3 main domains: staffing, organization, and services. Possible total scores range from a low of 15 to a high of 75, with high numbers representing better fidelity. Total scores below 56 are considered “not supported employment,” scores between 56 and 65 are “fair” implementation, and scores over 65 are “good” implementation. The total score for CMHC A was 60, and that for CMHC B was 66, indicating fair and good implementation, respectively.

Cognitive Training Program

Clients assigned to the CT + SE program participated in the Thinking Skills for Work Program, as described above. Following randomization to CT + SE, clients were contacted by their cognitive training specialist who initiated the assessment and computer cognitive training exercises of the program.

Measures

Diagnostic and Background Information

Psychiatric and substance use diagnoses were drawn from the clients' charts based on DSM-IV criteria.34 Employment history and receipt of vocational services were based on client interviews, supplemented by chart reviews and staff reports. Other background information such as educational level and other demographic characteristics were drawn from client interviews.

Employment Outcomes

Work outcomes, including the type of each job, hours worked, wages earned, and job tenure were assessed through regular contacts with clients and vocational staff members. All work obtained was competitive, as defined by the Substance Abuse and Mental Health Services Administration Employment Demonstration project,16 including work paying minimum wage or higher, jobs “owned” by the client (and not the agency), jobs not set aside for a person with a disability, and work that is integrated in the community.

Cognitive Functioning

A broad range of cognitive functions was assessed at baseline and 3-month follow-up, including attention and concentration, psychomotor speed, learning and memory, and executive functions. The following tests were employed in the cognitive battery:

Premorbid Academic Achievement

The Wide Range Achievement Test—III (WRAT-III) Reading subtest was used to measure premorbid academic achievement.35 This instrument measures word recognition reading performance. Performance on this test has been shown to be relatively preserved in schizophrenia, providing an index of premorbid educational attainment.36 The WRAT-III measure is the total score for words read correctly, converted to the grade-equivalent score. This measure was only administered at baseline.

Attention

Immediate attention was measured with the Digit Span (Wechsler Adult Intelligence Scale—Revised).38 Subjects are given a number string and asked to repeat it in the same order of presentation (Digit Span Forward) or backward (Digit Span Backwards). The measure of interest is the number correct for each condition.

Psychomotor Speed

Psychomotor speed was measured with Trail Making Test, Part A.37 This test is a timed measure of visual scanning ability and psychomotor speed that requires subjects to connect numbers in order. The measure used is the time, in seconds, to complete the task.

Information Processing Speed

Information processing speed was assessed with the Digit Symbol Substitution Test (DSST) from the Weschler Adult Intelligence Scale—Revised.38 For the DSST, subjects are asked to copy unique symbols below individual numbers (1–9) for 120 seconds. Number of symbols accurately copied is totaled and is the dependent variable.

Verbal Learning and Memory

Verbal learning and memory were assessed with the California Verbal Learning Test (CVLT).39 The CVLT involves the repeated presentation of a word list that consists of common items that are semantically related to 4 common conceptual categories (food, clothing, spices, or tools). The measures of interest are acquisition, determined by the total words recalled during the 5 acquisition trials (CVLT 1–5), and retention, determined by the total words recalled in the long-delay free recall, which occurs 20 minutes after the last acquisition trial.

Executive Functioning

Executive function was assessed with the Trail Making Test, Part B, and the Wisconsin Card Sorting Test (WCST).40 Trail Making Part B is similar to Part A but is a more challenging task because it requires subjects to connect consecutively numbered and lettered circles by alternating between the 2 sequences. The measure used is the number of seconds to complete the test. The WCST is a commonly used test of executive functioning that measures cognitive flexibility and problem-solving skills. Subjects are asked to match a series of cards to a set of 4 target stimuli, which are also cards. Subjects are provided with feedback on an item-by-item basis after they sort each of the item cards. After they determine 1 of the correct dimensions, referred to as “Categories,” 10 correct responses are required before the correct category is shifted to the next. Continued matching to a category that is no longer correct is considered a perseverative error. The variables of interest are the number of categories achieved and the percent of perseverative errors.

Overall Cognitive Functioning

A composite measure of overall cognitive functioning (not including premorbid intelligence measured on the WRAT) was computed by standardizing each of the cognitive measures (i.e., computing z-scores) and summing those scores separately for the baseline and follow-up assessments.

Psychopathology

Psychiatric symptoms were assessed with interviews conducted using the Positive and Negative Syndrome Scale (PANSS)41 pertaining to the prior week of functioning. Interviews were conducted by a trained research interviewer. Outcomes on the PANSS were analyzed using the 5-factor solution described by White and colleagues,42 which includes the following subscales: Positive, Negative, Depression, Autistic Preoccupation, and Activation. Regular reliability checks were conducted to ensure good inter-rater reliability.

Research Attrition and Program Exposure

Efforts were made to obtain follow-up employment data for the 1 year following randomization, regardless of program retention. Among the 44 randomized clients, full 12-month employment data were available on 32 (73%) clients. The average number of months of follow-up data on the sample was 10.91 months for the CT + SE group and 11.45 months for the SE Only group, which does not differ significantly (p > .1). Some clients dropped out of supported employment services and either transferred to other vocational services or stopped receiving vocational services altogether. The number of months receiving supported employment services was 9.69 for CT + SE and 9.53 for SE Only, which does not differ significantly (p > .1).

Exposure to the cognitive training program was defined as participating in a minimum of 6 computer cognitive training sessions. Six was chosen as the minimum number of sessions because the training exercises cover the full range of cognitive domains within the first 6 sessions, with subsequent sessions focusing on repeated practice of exercises across the domains. It should be noted that although we defined exposure to the program as participating in a minimum of 6 computer cognitive training sessions, clients in the CT + SE group who participated in fewer such sessions may nevertheless have benefited from receiving other parts of the program, such as assessment, job search planning, and consultative job support. Among the 23 people assigned to CT + SE, 21 (91%) were exposed to the program. The average number of computer cognitive training sessions completed was 19.6 (SD = 6.95, range: 4–24), which were completed over an average of 13.6 weeks (SD = 5.98, range: 2–28).

Statistical Analyses

Intent-to-treat analyses of the employment outcomes were conducted on the entire randomized sample who had some follow-up data (N = 44). Because the vast majority of clients assigned to the CT + SE program were exposed to the CT component, separate treatment exposure analyses were not performed. Although rates of follow-up were comparable between the 2 groups, 12-month follow-up data were not available for all clients. Therefore, we examined hours worked, wages earned, and number of jobs held cumulatively over the 1-year follow-up period and wages and hours per month of available follow-up.

The statistical analyses were divided into 4 steps. First, we compared the 2 sites on demographic characteristics, diagnoses, work history, baseline cognitive functioning, and baseline psychopathology using χ2 tests (for categorical variables) and t-tests (for continuous variables). Second, we compared the CT + SE group with the SE Only group on these same variables, using the same approaches. Third, we compared the CT + SE and SE Only programs on changes in cognitive functioning and psychopathology from baseline to the 3-month follow-up assessment with analyses of covariance (ANCOVA), with the follow-up cognitive or psychopathology assessment point as the dependent variable, the group as the independent variable, and the baseline measure as the covariate.

Last, we compared work outcomes between the 2 groups over the 1-year follow-up period. We compared the overall rates of work (work/no work) between the 2 groups by conducting a χ2 analysis. To evaluate whether treatment program interacted with site, we first conducted ANOVAs, predicting the continuous work outcomes from treatment group, site, and their interaction. Most work data were skewed (except number of jobs), so log transformations were used to normalize these data. Because even the log-transformed data were skewed, we also computed Mann Whitney U-tests to compare the treatment groups on the work outcomes.

Results

Comparison of Sites at Baseline

Clients recruited for the project at CMHC A differed in a number of background and baseline characteristics from clients recruited from CMHC B. Clients at CMHC A were significantly less likely to be married (χ2 = 7.50, df = 1, p = .006), were younger (t[42] = 4.10, p = .000), were more likely to have schizophrenia or schizoaffective disorder (χ2 = 6.23, df = 1, p = .044), were less likely to have a history of drug use disorder (χ2 = 5.98, df = 1, p = .014) and alcohol use disorder (χ2 = 3.87, df = 1, p = .049), were less likely to have a work history (χ2 = 7.13, df = 1, p = .008), were hospitalized at a younger age (t[37] = 3.05, p = .008), and had their most recent hospitalization at a younger age (t[37] = 2.56, p = .015) than clients at CMHC B (see table 1).

In addition, there were 2 differences between the sites in performance on the cognitive battery. Clients at CMHC A performed significantly better on Digit Span Backwards (Ms = 5.17 and 4.20, SDs = 1.67 and 1.14, respectively; t[42] = 2.02, p = .05) but worse on Trail Making Part A than clients at CMHC B (Ms = 46.66 and 37.07, SDs = 18.31 and 9.32, respectively; t[41] = 2.28, p = .028). There were no differences between the groups on any of the PANSS subscales.

Because of these site differences, we included site as an independent variable in all the analyses evaluating the effects of the treatment on neurocognitive, symptom, and work outcomes. Only 1 site–treatment interaction effect was significant (on number of jobs); therefore, for the other analyses we report effects that do not include site in the analysis as an independent variable.

Comparison of Groups at Baseline

The CT + SE group differed significantly (p > .05) from the SE Only group on only 1 demographic, diagnostic, work history, or illness history variable: years of education. Clients in the CT + SE group had significantly more years of education (M = 11.80, SD = 1.77) than the SE Only group (M = 10.76, SD = 1.64; t[42] = 2.02, p = .05). The 2 groups did not differ at baseline on any of the cognitive measures or on the PANSS subscales, indicating that the 2 groups were similar on almost all characteristics at baseline. To explore whether the difference in years of education between the 2 groups influenced the study findings, we ran parallel analyses including and not including education as a covariate. Years of education was related to almost no study outcome variables, and its inclusion as a covariate did not alter any results. Therefore, we report the results of analyses that do not include education as a covariate.

Cognitive Functioning and Psychopathology Outcomes

The results of the ANCOVAs evaluating the effects of the 2 programs on cognitive functioning and symptoms at 3 months are summarized in tables 2–3. Inspection of table 2 indicates that clients in the CT + SE program demonstrated significantly greater improvement in Trail Making Part B, CVLT 1–5, and the overall cognitive composite score than clients in SE Only. Table 3 shows that clients in CT + SE also improved significantly more in depression and autistic preoccupation on the PANSS than clients in SE Only.

Work Outcomes

A χ2 analysis comparing cumulative rates of work between the 2 groups indicates that significantly more clients in CT + SE worked (69.6%) than those in the SE Only program (4.8%; χ2[1] = 19.11, p = .000). Competitive work outcomes over 1 year for the 2 groups are summarized in table 4. Inspection of this table reveals that the CT + SE group held significantly more jobs, worked more hours, and earned more wages than the SE Only group.

Exploratory ANOVAs examining site–group interactions on the work outcomes indicate 1 significant interaction involving number of jobs held (F[1,40] = 4.70, p =.036). This effect, which is summarized in figure 1, indicates that although the CT + SE program was more effective than SE Only at both sites, the effect was significantly more robust at the CMHC B than the CMHC A site.

Discussion

The results of this study provide support for the feasibility of CT + SE (the Thinking Skills for Work Program) for helping clients with severe mental illness enrolled in supported employment. The overall rate of retention in the program was high; 91% of the clients who agreed to participate were successfully engaged and completed a minimum of 6 cognitive training sessions. Furthermore, the program was successfully run in 2 supported employment programs serving inner-city minority clients, suggesting that it can be implemented in “real-world” settings where people receive mental health and vocational services.

The study also provides preliminary support for the effectiveness of the Thinking Skills for Work Program at improving cognitive functioning and employment outcomes. Clients who participated in the program showed significantly greater improvements in several areas of cognitive functioning over the first 3 months of the study, as well as in the overall composite cognitive score and Autistic Preoccupation subscale on the PANSS compared to those who received supported employment services alone. Finally, clients who participated in the Thinking Skills for Work Program had substantially higher rates of competitive work, including number of jobs, hours worked, and wages earned, suggesting that the program was successful at helping many clients achieve their goals of competitive employment.

An intriguing finding is that clients who participated in the Thinking Skills for Work Program demonstrated significantly greater improvements on the Depression subscale of the PANSS compared to clients who received supported employment only. Several different factors could account for these findings. It is possible that cognitive training itself reduces depression because of the reinforcement inherent in successfully completing increasingly more difficult computer exercises. Several other studies have shown that participation in cognitive training is associated with significant improvements in depression,43 distress,44 and self-esteem.45–46 However, other studies have not reported this effect,47–48 and Wykes and colleagues have found that the beneficial effects of cognitive training on self-esteem were not sustained after training ended.49

Another possibility is that improvements in depression were related, either wholly or in part, to the superior work outcomes of the clients in the Thinking Skills for Work Program. Consistent with the importance of work to persons with severe mental illness,50 obtaining competitive work is associated with a range of positive emotions in people, including improved mood, self-esteem, and life satisfaction.5, 7–8, 51 Alternatively, participation in cognitive training and obtaining work may have operated in concert to improve mood and self-confidence. Considering that symptoms were assessed at only 3 months after participation in the program, there is a need to evaluate the longer-term effects of the Thinking Skills for Work Program on mood.

The work outcomes of the SE Only group were quite low despite the fact that both agencies had supported employment programs that were integrated with clinical services. The low rate of work in these programs cannot be directly compared with work rates in prior randomized controlled trials of supported employment19 because of differences in eligibility criteria. Prior controlled studies of supported employment in people with severe mental illness have chiefly focused on clients who are unemployed and want to work, and they report cumulative work rates in the range of 60 to 80% over a 1.5- to 2-year period. The present study focused on clients who had specifically experienced a job failure (defined as either being fired or walking off the job without another job lined up) either when receiving vocational services or not. These selection criteria were used in order to evaluate whether the Thinking Skills for Work Program was specifically effective for clients with a history of job failures. The very low rate of competitive work among clients who received supported employment only suggests that these selection criteria were effective at identifying a subgroup of clients who were less likely to benefit from supported employment alone.

There is some evidence that the Thinking Skills for Work Program was more effective at 1 site than the other in helping clients obtain jobs (figure 1), and this difference could not be accounted for by any client differences between the agencies. The higher fidelity to the supported employment model at CMHC B than at CMHC A could account for some of this difference if the cognitive training and supported employment programs act synergistically with one another, although the magnitude of the difference in fidelity was not great (66 for CMHC A, 60 for CMHC B). Another possibility could be differences in involvement in day treatment between the 2 centers. Clients at CMHC A were active participants in a day treatment program, whereas clients at CMHC B were drawn from a variety of different psychiatric and rehabilitation services throughout the agency. It is possible that when day treatment is available it may compete with the goal of obtaining work in supported employment programs by providing other meaningful activities in which clients can participate. However, this interpretation should be tempered by the fact that there were no interactions between site and treatment group for other work outcomes, including the hours worked and wages earned.

The Thinking Skills for Work Program involves multiple components, including comprehensive cognitive and vocational assessment, computer cognitive skills training, individualized job planning, and consultative job support with a supported employment specialist. As the goal of the program is to integrate the training of cognitive skills and the development of compensatory strategies for minimizing the effects of cognitive impairment on work into ongoing supported employment services, with the present study design it is not possible to know how the program works or which components are essential and which are not. However, several conjectures may provide useful avenues for future investigation.

First, the focus of the program was on helping clients get and keep competitive jobs, and clients who had experienced difficulties working were specifically selected with this goal in mind. The clear focus of the program on work ensured that participants had motivation to pursue this goal, a critical criterion for cognitive remediation.28, 52 The emphasis of the Thinking Skills for Work Program on a simple and easily measurable outcome differs from many prior studies of cognitive remediation that have focused on broad areas of functioning such as symptoms,43, 47, 48, 53, 54 social functioning,32, 49, 55, 56 or global functioning57, 58 or in which no functional outcomes are identified.44, 48, 59–63 Specifying readily measurable outcomes also makes it easier to track the effectiveness of the program when it is being delivered, which can enable the rehabilitation providers to tailor their approaches when necessary to optimize outcomes.

Second, there is much debate in the cognitive rehabilitation literature as to the specific training methods that should be employed (e.g., computer-based versus individual or group exercises; practice drills and/or strategy coaching) and whether compensatory strategies for coping with cognitive impairments should be included as a focus of training, with some programs emphasizing the former, others involving the latter, and still others providing a blend of both skills.29, 64–67 The practical need for developing some compensatory strategies to minimize the effects of cognitive impairment on functioning is underscored by the results of reviews of cognitive remediation studies (as well as the results of this study; see table 2), which have concluded that although improvements in cognitive performance are evident, some degree of impairment typically persists.29, 68 Indeed, Krabbendam and Aleman's review of cognitive remediation studies concludes that there was more support for teaching compensatory strategies over remediative ones.28 Thus, another key to the positive effects of the Thinking Skills for Work Program is its multimodal nature that includes both computer cognitive training approaches and individualized strategies for either overcoming or compensating for the effects of cognitive impairment at the workplace. Silverstein and Wilkniss53 have commented on the importance of individualizing cognitive rehabilitation programs in order to ensure that training is tailored to the specific needs of the client.

Third, the Thinking Skills for Work Program is integrated with supported employment, an evidence-based practice for improving vocational functioning in persons with severe mental illness.19 The integration of cognitive training with supported employment provides opportunities for maximizing the transfer of cognitive skills practiced in the computer training exercises to the workplace and for identifying and developing coping strategies for managing cognitive impairments that interfere with work performance. Furthermore, since supported employment is not a time-limited intervention, the availability of ongoing supports may provide an answer to the question posed by Wykes and colleagues of how gains in cognitive functioning can be maintained after the initial training phase.46 Although some other programs have integrated cognitive remediation with established rehabilitation approaches, most notably social skills training,32, 69 most of this work has focused on inpatient populations where the impact of these interventions on social functioning remains unclear. Thus, the integration of cognitive training with supported employment here may have contributed to the positive outcomes in competitive work.

A number of limitations of this study should be noted. First, although employment data were collected over 1 year, follow-up assessments of cognitive functioning and symptoms were only conducted at 3 months, leaving unanswered the question of whether the cognitive gains and improved depression were sustained over time. Second, the relatively small sample size precluded exploratory analyses of cognitive changes and work within the group of clients who participated in the cognitive training program. Third, the fidelity of the 2 programs to the supported employment model was moderate, suggesting that work rates for the SE Only group would be higher in programs with stronger fidelity but raising the question of whether work rates would be even higher in the CT + SE program. Research is needed to evaluate the effects of the Thinking Skills for Work Program in programs with higher fidelity to supported employment. A final limitation was the cognitive battery employed, which lacked an assessment of sustained attention and a more refined measure of working memory, cognitive areas that are improved by cognitive training54, 70 and associated with competitive work.27, 71, 72

These limitations notwithstanding, the present findings provide encouraging support for the feasibility of the Thinking Skills for Work Program and its potential for helping clients with severe mental illness achieve their vocational goals. Furthermore, this study showed that the program can be successfully implemented in challenging inner-city settings with minority clients who have relatively low levels of educational attainment. Additional research is warranted to replicate and extend the effects of this program.

Table 1.

Demographic and Diagnostic Differences of Participants by Community Mental Health Center

 CMHC A (N = 29)
 
CMHC B (N = 15)
 
Variable N N 
Categorical Variables     
Treatment Group     
 CT + SE 16 55 47 
 SE Only 13 45 53 
Gender     
 Male 18 62 40 
 Female 11 38 60 
Ethnicity     
 African American 17 59 13 87 
 Hispanic 24 
 Caucasian 14 13 
 Asian 
Marital Status     
 Never Married 28 97 10 67 
 Ever Married 33 
Diagnosis     
 Schizophrenia 22 76 10 67 
 Schizoaffective 
 Mood Disorder 17 33 
Current Alcohol Use Disorder     
 No 26 90 14 93 
 Yes 10 
Lifetime Alcohol Use Disorder     
 No 25 86 40 
 Yes 14 60 
Current Drug Use Disorder     
 No 27 93 14 93 
 Yes 
Lifetime Drug Use Disorder     
 No 10 65 11 73 
 Yes 19 35 27 
Continuous Variables Mean (SD) Mean (SD) 
Age 31.4 (12.4) 43.7 (7.47) 
Years of Education 11.5 (1.90) 10.9 (1.46) 
Wide Range of Achievement Test 
Age at First Hospitalization 19.4 (6.16) 29.1 (10.19) 
Age at Last Hospitalization 29.6 (12.34) 39.6 (7.93) 
Months Since Last Job 48.47 (35.74) 30.3 (30.26) 
Months of Longest Job 22.9 (32.81) 21.5 (12.06) 
Hours Worked Past Year 39.73 (83.91) 20.69 (46.44) 
 CMHC A (N = 29)
 
CMHC B (N = 15)
 
Variable N N 
Categorical Variables     
Treatment Group     
 CT + SE 16 55 47 
 SE Only 13 45 53 
Gender     
 Male 18 62 40 
 Female 11 38 60 
Ethnicity     
 African American 17 59 13 87 
 Hispanic 24 
 Caucasian 14 13 
 Asian 
Marital Status     
 Never Married 28 97 10 67 
 Ever Married 33 
Diagnosis     
 Schizophrenia 22 76 10 67 
 Schizoaffective 
 Mood Disorder 17 33 
Current Alcohol Use Disorder     
 No 26 90 14 93 
 Yes 10 
Lifetime Alcohol Use Disorder     
 No 25 86 40 
 Yes 14 60 
Current Drug Use Disorder     
 No 27 93 14 93 
 Yes 
Lifetime Drug Use Disorder     
 No 10 65 11 73 
 Yes 19 35 27 
Continuous Variables Mean (SD) Mean (SD) 
Age 31.4 (12.4) 43.7 (7.47) 
Years of Education 11.5 (1.90) 10.9 (1.46) 
Wide Range of Achievement Test 
Age at First Hospitalization 19.4 (6.16) 29.1 (10.19) 
Age at Last Hospitalization 29.6 (12.34) 39.6 (7.93) 
Months Since Last Job 48.47 (35.74) 30.3 (30.26) 
Months of Longest Job 22.9 (32.81) 21.5 (12.06) 
Hours Worked Past Year 39.73 (83.91) 20.69 (46.44) 

Note: CMHC = Community Mental Health Center, CT + SE = group receiving cognitive training and supported employment, SE Only = group receiving supported employment only.

Table 2.

Changes in Cognitive Functioning From Baseline to 3 Months for Cognitive Training and Supported Employment (CT + SE) and Supported Employment–Only (SE Only) Groups

  CT + SE
 
SE Only
 
   
Instrument Time Mean (SD) Mean (SD) F-Test df p 
Digit Span Wechsler Adult Intelligence Scale—III 
 Forward baseline 7.05 (1.90) 6.83 (2.01)    
 3 months 7.84 (2.36) 7.28 (2.78) 0.32 1,34 .575 
 Backward baseline 5.17 (1.54) 4.50 (1.65)    
 3 months 5.83 (2.38) 4.61 (1.54) 7.78 1,33 .19 
Digit Symbol baseline 38.27 (12.50) 39.94 (6.85)    
 Wechsler Adult Intelligence Scale—III 3 months 41.39 (13.51) 41.55 (8.20) 0.31 1,33 .582 
Trail Making         
 Part A baseline 47.55 (17.68) 42.86 (15.07)    
 3 months 43.61 (14.68) 46.89 (21.31) 1.44 1,33 .238 
 Part B baseline 150.78 (80.75) 136.67 (65.74)    
 3 months 107.28 (54.83) 144.39 (78.28) 12.95 1,33 .002 
California Verbal Learning Test         
 Trials 1–5 baseline 34.88 (12.59) 34.35 (13.30)    
 3 months 44.05 (9.91) 38.65 (11.93) 4.28 1,32 .047 
 Long-Delay Free Recall baseline 7.73 (3.40) 7.06 (3.58)    
 3 months 9.42 (3.92) 8.00 (3.76) 0.91 1,33 .348 
Wisconsin Card Sorting Test         
 Total Categories baseline 2.00 (2.23) 1.64 (1.44)    
 3 months 2.63 (2.45) 2.07 (2.02) 0.21 1,30 .650 
 % Perseverative Errors baseline 38.20 (21.23) 33.91 (10.34)    
 3 months 30.16 (20.85) 31.93 (12.92) 0.49 1,26 .491 
Composite Cognition Score baseline −0.04 (0.64) 0.03 (0.60)    
 3 months 0.14 (0.70) −0.16 (0.64) 9.63 1,34 .005 
  CT + SE
 
SE Only
 
   
Instrument Time Mean (SD) Mean (SD) F-Test df p 
Digit Span Wechsler Adult Intelligence Scale—III 
 Forward baseline 7.05 (1.90) 6.83 (2.01)    
 3 months 7.84 (2.36) 7.28 (2.78) 0.32 1,34 .575 
 Backward baseline 5.17 (1.54) 4.50 (1.65)    
 3 months 5.83 (2.38) 4.61 (1.54) 7.78 1,33 .19 
Digit Symbol baseline 38.27 (12.50) 39.94 (6.85)    
 Wechsler Adult Intelligence Scale—III 3 months 41.39 (13.51) 41.55 (8.20) 0.31 1,33 .582 
Trail Making         
 Part A baseline 47.55 (17.68) 42.86 (15.07)    
 3 months 43.61 (14.68) 46.89 (21.31) 1.44 1,33 .238 
 Part B baseline 150.78 (80.75) 136.67 (65.74)    
 3 months 107.28 (54.83) 144.39 (78.28) 12.95 1,33 .002 
California Verbal Learning Test         
 Trials 1–5 baseline 34.88 (12.59) 34.35 (13.30)    
 3 months 44.05 (9.91) 38.65 (11.93) 4.28 1,32 .047 
 Long-Delay Free Recall baseline 7.73 (3.40) 7.06 (3.58)    
 3 months 9.42 (3.92) 8.00 (3.76) 0.91 1,33 .348 
Wisconsin Card Sorting Test         
 Total Categories baseline 2.00 (2.23) 1.64 (1.44)    
 3 months 2.63 (2.45) 2.07 (2.02) 0.21 1,30 .650 
 % Perseverative Errors baseline 38.20 (21.23) 33.91 (10.34)    
 3 months 30.16 (20.85) 31.93 (12.92) 0.49 1,26 .491 
Composite Cognition Score baseline −0.04 (0.64) 0.03 (0.60)    
 3 months 0.14 (0.70) −0.16 (0.64) 9.63 1,34 .005 

Note: Treatment group effect (CT + SE versus SE Only) based on analysis of covariance with baseline score as covariate.

Table 3.

Changes in Positive and Negative Syndrome Scale Scores From Baseline to 3 Months for Cognitive Training and Supported Employment (CT + SE) and Supported Employment–Only (SE Only) Groups

  CT + SE
 
SE Only
 
   
Subscale Time Mean (SD) Mean (SD) F-Test df p 
Negative baseline 2.19 (0.54) 2.26 (0.78)    
 3 months 2.03 (0.61) 2.24 (0.51) 1.31 1,30 .261 
Positive baseline 1.71 (0.86) 1.51 (0.59)    
 3 months 1.79 (0.82) 1.75 (0.69) 0.42 1,31 .522 
Activation baseline 1.42 (0.45) 1.61 (0.81)    
 3 months 1.49 (0.48) 1.36 (0.42) 1.53 1,31 .226 
Depression baseline 2.31 (0.56) 2.15 (0.73)    
 3 months 2.15 (0.67) 2.46 (0.81) 4.69 1,31 .038 
Autistic Preoccupation baseline 1.47 (0.28) 1.49 (0.37)    
 3 months 1.38 (0.31) 1.56 (0.22) 4.73 1,31 .037 
  CT + SE
 
SE Only
 
   
Subscale Time Mean (SD) Mean (SD) F-Test df p 
Negative baseline 2.19 (0.54) 2.26 (0.78)    
 3 months 2.03 (0.61) 2.24 (0.51) 1.31 1,30 .261 
Positive baseline 1.71 (0.86) 1.51 (0.59)    
 3 months 1.79 (0.82) 1.75 (0.69) 0.42 1,31 .522 
Activation baseline 1.42 (0.45) 1.61 (0.81)    
 3 months 1.49 (0.48) 1.36 (0.42) 1.53 1,31 .226 
Depression baseline 2.31 (0.56) 2.15 (0.73)    
 3 months 2.15 (0.67) 2.46 (0.81) 4.69 1,31 .038 
Autistic Preoccupation baseline 1.47 (0.28) 1.49 (0.37)    
 3 months 1.38 (0.31) 1.56 (0.22) 4.73 1,31 .037 

Note: Treatment group effect (CT + SE versus SE Only) based on analysis of covariance with baseline score as covariate.

Table 4.

Competitive Work Outcomes Over 1 Year for Cognitive Training and Supported Employment (CT + SE) and Supported Employment–Only (SE Only) Groups

 CT + SE
 
SE Only
 
Mann-Whitney U-Test 
Work Outcome N Mean (SD) N Mean (SD)  
Total Jobs 23 0.96 (0.82) 21 0.05 (0.22) 4.32 
Total Hours 23 379.91 (419.14) 21 30.95 (141.84) 4.11 
Total Wages 23 2,207.91 (3,166.11) 21 182.05 (834.24) 4.09 
Hours/Month 23 34.48 (44.48) 21 2.58 (11.82) 4.14 
Wages/Month 23 199.11 (270.00) 21 15.17 (69.52) 4.11 
 CT + SE
 
SE Only
 
Mann-Whitney U-Test 
Work Outcome N Mean (SD) N Mean (SD)  
Total Jobs 23 0.96 (0.82) 21 0.05 (0.22) 4.32 
Total Hours 23 379.91 (419.14) 21 30.95 (141.84) 4.11 
Total Wages 23 2,207.91 (3,166.11) 21 182.05 (834.24) 4.09 
Hours/Month 23 34.48 (44.48) 21 2.58 (11.82) 4.14 
Wages/Month 23 199.11 (270.00) 21 15.17 (69.52) 4.11 

p = .000.

Fig. 1.

Number of Jobs Obtained for Clients in Cognitive Training and Supported Employment (CT + SE) and Supported Employment–Only (SE Only) Programs for Community Mental Health Centers A and B.

Fig. 1.

Number of Jobs Obtained for Clients in Cognitive Training and Supported Employment (CT + SE) and Supported Employment–Only (SE Only) Programs for Community Mental Health Centers A and B.

This research was supported by a grant from the New York Work Exchange Coalition of Voluntary Mental Health Agencies, Inc. We appreciate the assistance of the following people for their help on the project: Karin Abrahamian, Sue Buchanan, Liza Cahill, Bill Carr, Anthony Cox, Charles Jones, Amy Telesford, Leslie Klein, Diana Little, Ignatius Low, Anita Powell, Paula Rivers, and Maje Waldo. Appreciation is also extended to Tamar Kairy for project coordination and Rosemarie Wolfe for assistance with data analysis.

References

1.
Anthony, WA and Blanch, A. Supported employment for persons who are psychiatrically disabled: an historical and conceptual perspective.
Psychosoc Rehabil J
 
1987
;
11
5
–23.
2.
Marwaha, S and Johnson, S. Schizophrenia and employment: a review.
Soc Psych Psych Epid
 
2004
;
39
337
–349.
3.
Mueser, KT, Salyers, MP, Mueser, PR. A prospective analysis of work in schizophrenia.
Schizophrenia Bull
 
2001
;
27
281
–296.
4.
Rogers, ES, Walsh, D, Masotta, L, Danley, K.
Massachusetts Survey of Client Preferences for Community Support Services (Final Report)
  Boston: Center for Psychiatric Rehabilitation
1991
.
5.
Arns, PG and Linney, JA. Work, self, and life satisfaction for persons with severe and persistent mental disorders.
Psychosoc Rehabil J
 
1993
;
17
63
–79.
6.
Bell, MD, Lysaker, PH, Milstein, RM. Clinical benefits of paid work activity in schizophrenia.
Schizophrenia Bull
 
1996
;
22
51
–67.
7.
Bond, GR, Resnick, SG, Drake, RE, Xie, H, McHugo, GJ, Bebout, RR. Does competitive employment improve nonvocational outcomes for people with severe mental illness?
J Consult Clin Psych
 
2001
;
69
489
–501.
8.
Mueser, KT, Becker, DR, Torrey, WC, et al. Work and nonvocational domains of functioning in persons with severe mental illness: a longitudinal analysis.
J Nerv Ment Dis
 
1997
;
185
419
–426.
9.
Bond, GR, Drake, RE, Mueser, KT, Becker, DR. An update on supported employment for people with severe mental illness.
Psychiatr Serv
 
1997
;
48
335
–346.
10.
Becker, DR and Drake, RE.
A Working Life for People With Severe Mental Illness
  New York: Oxford University Press
2003
.
11.
Bond, G and Dincin, J. Accelerating entry into transitional employment in a psychosocial rehabilitation agency.
Rehabil Psychol
 
1986
;
31
143
–155.
12.
Drake, RE, McHugo, GJ, Becker, DR, Anthony, WA, Clark, RE. The New Hampshire study of supported employment for people with severe mental illness: vocational outcomes.
J Consult Clin Psych
 
1996
;
64
391
–399.
13.
Drake, RE, McHugo, GJ, Bebout, RR, et al. A randomized clinical trial of supported employment for inner-city patients with severe mental illness.
Arch Gen Psychiat
 
1999
;
56
627
–33.
14.
Gervey, R and Bedell, JR. Supported employment in vocational rehabilitation. In Bedell, JR (Ed.).
Psychological Assessment and Treatment of Persons With Severe Mental Disorders
  Washington, D.C.: Taylor and Francis
1994
pp.
139
–163.
15.
Gold PB, Meisler N, Santos AB, Williams OH, Carnemolla MA, Kelleher J. Randomized trial of supported employment integrated with assertive community treatment in the rural South: employment outcomes for persons with severe mental illness. 2004;under review.
16.
Lehman, AF, Goldberg, R, Dixon, LB, et al. Improving employment outcomes for persons with severe mental illnesses.
Arch Gen Psychiat
 
2002
;
59
165
–172.
17.
McFarlane, WR, Dushay, RA, Deakins, SM, et al. Employment outcomes in family-aided assertive community treatment.
Am J Orthopsychiat
 
2000
;
70
203
–214.
18.
Mueser, KT, Clark, RE, Haines, M, et al. The Hartford study of supported employment for severe mental illness.
J Consult Clin Psych
 
2004
;
72
479
–490.
19.
Bond, GR, Becker, DR, Drake, RE, et al. Implementing supported employment as an evidence-based practice.
Psychiatr Serv
 
2001
;
52
313
–322.
20.
Shafer, MS and Huang, HW. The utilization of survival analysis to evaluate supported employment services.
J Vocat Rehabil
 
1995
;
5
103
–113.
21.
McGurk, SR, Mueser, KT, Harvey, PD, Marder, J, LaPuglia, R. Cognitive and clinical predictors of work outcomes in clients with schizophrenia.
Psychiatr Serv
 
2003
;
54
1129
–1135.
22.
Wong, KK, Chiu, L-p, Tang, S-w, et al. A supported employment program for people with mental illness in Hong Kong.
Am J Psychiat Rehabil
 
2004
;
7
83
–96.
23.
Becker, DR, Drake, RE, Bond, GR, Xie, H, Dain, BJ, Harrison, K. Job terminations among persons with severe mental illness participating in supported employment.
Community Ment Hlt J
 
1998
;
34
71
–82.
24.
Mueser, KT, Becker, DR, Wolfe, R. Supported employment, job preferences, and job tenure and satisfaction.
J Ment Health
 
2001
;
10
411
–417.
25.
Wykes, T and Dunn, G. Cognitive deficit and the prediction of rehabilitation success in a chronic psychiatric group.
Psychol Med
 
1992
;
22
389
–398.
26.
Sharma, T and Harvey, P.
Cognition in Schizophrenia: Impairments, Importance and Treatment Strategies
  New York: Oxford University Press
2000
.
27.
McGurk, SR and Mueser, KT. Cognitive functioning, symptoms, and work in supported employment: a review and heuristic model.
Schizophr Res
 
2004
;
70
147
–174.
28.
Krabbendam, L and Aleman, A. Cognitive rehabilitation in schizophrenia: a quantitative analysis of controlled studies.
Psychopharmacology
 
2003
;
169
376
–382.
29.
Twamley, EW, Jeste, DV, Bellack, AS. A review of cognitive training in schizophrenia.
Schizophrenia Bull
 
2003
;
29
359
–382.
30.
Pilling, S, Bebbington, P, Kuipers, E, et al. Psychological treatments in schizophrenia: II. meta-analyses of randomized controlled trials of social skills training and cognitive remediation.
Psychol Med
 
2002
;
32
783
–791.
31.
Hogarty, GE, Flesher, S, Ulrich, RF, et al. Cognitive enhancement therapy for schizophrenia: effects of a 2-year randomized trial on cognition and behavior.
Arch Gen Psychiat
 
2004
;
61
866
–876.
32.
Spaulding, WD, Reed, D, Sullivan, M, Richardson, C, Weiler, M. Effects of cognitive treatment in psychiatric rehabilitation.
Schizophrenia Bull
 
1999
;
25
657
–676.
33.
Bond, GR, Becker, DR, Drake, RE, Vogler, KM. A fidelity scale for the individual placement and support model of supported employment.
Rehabil Couns Bull
 
1997
;
40
265
–284.
34.
American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 4th ed. Washington, D.C.: American Psychiatric Association .
35.
Wilkinson, G.
Wide Range Achievement Test 3 (Manual)
  Wilmington, Del.: Wide Range Inc.
1993
.
36.
Harvey, PD, Moriarty, PJ, Friedman, JI, et al. Differential preservation of cognitive functions in geriatric patients with lifelong chronic schizophrenia: less impairment in reading compared to other skill areas.
Biol Psychiat
 
2000
;
47
962
–968.
37.
Radford, LM, Chaney, EF, O'Leary, MR. Screening for cognitive impairment among inpatients.
J Clin Psychiat
 
1978
;
39
712
.
38.
Wechsler, D.
Wechsler Adult Intelligence Scale—Revised
  New York: Harcourt, Brace, Jovanovich
1981
.
39.
Delis, DC, Kramer, JH, Kaplan, E, Ober, BA.
California Verbal Learning and Memory Test (Manual)
  San Antonio: Psychological Corporation
1987
.
40.
Berg, EA. A simple objective test for measuring flexibility in thinking.
J Gen Psychol
 
1948
;
39
15
–22.
41.
Kay, SR, Opler, LA, Fiszbein, A. The Positive and Negative Syndrome Scale (PANSS) for schizophrenia.
Schizophrenia Bull
 
1987
;
13
261
–276.
42.
White, L, Harvey, PD, Parella, M, Knobler, H, Powchik, P, Davidson, M. Empirical assessment of the factorial structure of clinical symptoms in schizophrenic patients: symptom structure in geriatric and nongeriatric samples.
New Trends Exp Clin Psychiat
 
1994
;
10
75
–83.
43.
Bark, N, Revheim, N, Huq, F, Khalderov, V, Ganz, ZW, Medalia, A. The impact of cognitive remediation on psychiatric symptoms of schizophrenia.
Schizophr Res
 
2003
;
63
229
–235.
44.
Medalia, A, Dorn, H, Watras-Gans, S. Treating problem solving deficits on an acute psychiatric inpatient unit.
Psychiat Res
 
2000
;
97
79
–88.
45.
Bender, S, Dittman-Balcar, A, Prehn, G, Peters, S, Thienel, R, Gastpar, M. Subjektives erleben eines computergestützten kognitiven trainings durch patienten mit schizophrenien? [How do patients with schizophrenia experience computer-based cognitive training?].
Nervenarzt
 
2004
;
75
44
–50.
46.
Wykes, T, Reeder, C, Corner, J, Williams, C, Everitt, B. The effects of neurocognitive remediation on executive processing in patients with schizophrenia.
Schizophrenia Bull
 
1999
;
25
291
–307.
47.
Bellucci, DM, Glaberman, K, Haslam, N. Computer-assisted cognitive rehabilitation reduces negative symptoms in the severely mentally ill.
Schizophr Res
 
2002
;
59
225
–232.
48.
Medalia, A, Aluma, M, Tryon, W, Merriam, AE. Effectiveness of attention training in schizophrenia.
Schizophrenia Bull
 
1998
;
24
147
–152.
49.
Wykes, T, Reeder, C, Williams, C, Corner, J, Rice, C, Everitt, B. Are the effects of cognitive remediation therapy (CTR) durable? results from an exploratory trial in schizophrenia.
Schizophr Res
 
2003
;
61
163
–174.
50.
Provencher, HP, Gregg, R, Mead, S, Mueser, KT. The role of work in recovery of persons with psychiatric disabilities.
Psychiatr Rehabil J
 
2002
;
26
132
–144.
51.
Bell, MD, Milstein, RM, Lysaker, PH. Pay and participation in work activity: clinical benefits for clients with schizophrenia.
Psychosoc Rehabil J
 
1993
;
17
173
–177.
52.
Silverstein, SM and Wilkniss, SM. At issue: the future of cognitive rehabilitation of schizophrenia.
Schizophrenia Bull
 
2004
;
30
679
–692.
53.
Fiszdon, JM, Bryson, GJ, Wexler, BE, Bell, MD. Durability of cognitive remediation training in schizophrenia: performance on two memory tasks at 6-month and 12-month follow-up.
Psychiat Res
 
2004
;
125
1
–7.
54.
Hermanutz, M and Gestrich, J. Computer-assisted attention training in schizophrenics: a comparative study.
Eur Arch Psy Clin N
 
1991
;
240
282
–287.
55.
Olbrich, R and Mussgay, L. Reduction of schizophrenic deficits by cognitive training: an evaluation study.
Eur Arch Psy Neurol Sci
 
1990
;
239
366
–369.
56.
Penadés, R, Boget, T, Catalán, R, Bernardo, M, Gastó, C, Salamero, M. Cognitive mechanisms, psychosocial functioning, and neurocognitive rehabilitation in schizophrenia.
Schizophr Res
 
2003
;
63
219
–227.
57.
Ueland, T and Rund, BR. A controlled randomized treatment study: the effects of a cognitive remediation program on adolescents with early onset psychosis.
Acta Psychiat Scand
 
2004
;
109
70
–74.
58.
Ueland, T and Rund, BR. Cognitive remediation for adolescents with early onset psychosis: a 1-year follow-up study.
Acta Psychiat Scand
 
2005
;
111
193
–201.
59.
Bellack, AS, Gold, JM, Buchanan, RW. Cognitive rehabilitation for schizophrenia: problems, prospects, and strategies.
Schizophrenia Bull
 
1999
;
25
257
–274.
60.
Lopez-Luengo, B and Vazquez, C. Effects of attention process training on cognitive functioning of schizophrenic patients.
Psychiat Res
 
2003
;
119
41
–53.
61.
Medalia, A, Revheim, N, Casey, M. Remediation of problem-solving skills in schizophrenia: evidence of a persistent effect.
Schizophr Res
 
2002
;
57
165
–171.
62.
Sartory, G, Zorn, C, Groetzinger, G, Windgassen, K. Computerized cognitive rehabilitation improves verbal learning and processing speed in schizophrenia.
Schizophr Res
 
2005
;
75
219
–223.
63.
van der Gaag, M, Kern, RS, van den Bosch, RJ, Liberman, RP. A controlled trial of cognitive remediation in schizophrenia.
Schizophrenia Bull
 
2002
;
28
167
–176.
64.
Bellack, AS. Cognitive rehabilitation for schizophrenia: is it possible? is it necessary?
Schizophrenia Bull
 
1992
;
18
43
–50.
65.
Kurtz, MM, Moberg, PJ, Gur, RC, Gur, RE. Approaches to cognitive remediation of neuropsychological deficits in schizophrenia: a review and meta-analysis.
Neuropsychol Rev
 
2001
;
11
197
–210.
66.
Liberman, RP. Keynote lecture: cognitive remediation in schizophrenia. In Kashima, H, Falloon, IRH, Mizuno, M, Asai, M (Eds.).
Comprehensive Treatment of Schizophrenia
  Tokyo: Springer-Verlag
2002
pp.
254
–275.
67.
Reeder, C, Newton, E, Frangou, S, Wykes, T. Which executive skills should we target to affect social functioning and symptom change? a study of a cognitive remediation therapy program.
Schizophrenia Bull
 
2004
;
30
87
–100.
68.
Wykes, T and van der Gaag, M. Is it time to develop a new cognitive therapy for psychosis—cognitive remediation therapy (CRT)?
Clin Psychol Rev
 
2001
;
21
1227
–1256.
69.
Brenner, H, Roder, V, Hodel, B, Kienzle, N, Reed, D, Liberman, R.
Integrated Psychological Therapy for Schizophrenic Patients
  Seattle: Hogrefe and Huber Publishers
1994
.
70.
Bell, MD, Bryson, G, Greig, T, Corcoran, C, Wexler, RE. Neurocognitive enhancement therapy with work therapy.
Arch Gen Psychiat
 
2001
;
58
763
–768.
71.
McGurk, SR and Meltzer, HY. The role of cognition in vocational functioning in schizophrenia.
Schizophr Res
 
2000
;
45
175
–184.
72.
McGurk, SR and Mueser, KT. Cognitive functioning and employment in severe mental illness.
J Nerv Ment Dis
 
2003
;
191
789
–798.