Abstract

Purpose: This study describes the creation and use of a web-based resource, designed to help nursing homes implement quality improvements through changes in staffing characteristics. Design and Methods: Information on staffing characteristics (i.e., staffing levels, turnover, stability, and use of agency staff), facility characteristics (e.g., ownership, size), and quality (i.e., quality measures [QMs]) coming from 2,946 nursing homes were utilized in a simulation analysis. The simulation was used to predict changes in each of 11 QMs, based on changes in staffing characteristics for individual nursing homes using the web-tool. Results: The website was visited 2,347 times in the first 6 months after it became operational. Evidence would suggest that it was useful in informing nursing home management of the importance of staffing and facilitating staffing changes; moreover, active users of the website showed improvements in quality, with an average of 5 of the 11 QMs improving by more than 1%. Implications: The web-based resource may be a feasible low-cost model that can be replicated, to provide further information for other areas of quality improvement in nursing homes.

A considerable amount of research in nursing homes has demonstrated that multiple staffing characteristics are important influences on quality of care. The research literature clearly shows that accounting for multiple staffing characteristics helps us understand the almost intransigent low quality observed in many nursing homes (Castle & Engberg, 2007; Kim, Harrington, & Greene, 2009; Kim, Kovner, Harrington, Greene, & Mezey, 2009). This is specifically true with respect to the staffing characteristics of turnover, staffing levels, stability, and agency staff use. It is important, therefore, to convey these findings to nursing home top management (i.e., Nursing Home Administrators [NHAs] and Directors of Nursing). This manuscript describes a project designed to convey the potential quality consequences of differences in staffing characteristics and quality improvements that can potentially occur, by altering these staffing characteristics.

Nursing homes have been noted as having poor staffing characteristics for several decades (Castle, 2008). Moreover, the research literature has consistently pointed out the folly of these poor staffing characteristics with respect to quality of care. For example, high staff turnover, low staffing levels, and high agency staff use have all being shown to be associated with high numbers of deficiency citations (Kim, Harrington, et al., 2009). Thus, it is likely that many nursing home top managers are cognizant that lower staff turnover, higher staffing levels, higher staff stability, and low agency staff use are advantageous. The fact that these staffing problems remain is almost certainly multifaceted and includes lack of a business case for improvements; problems with top management such as high turnover; and inadequate reimbursement (Hyer, Temple, & Johnson, 2009).

Nursing homes are seldom in a position to widely alter their current staffing characteristics and often exhibit multiple staffing issues (e.g., high staff turnover, low staffing levels, and high agency staff use). Thus, commonly quoted information about optimal staffing may be of little practical use for many nursing homes. Research reports do not address the question of what individual facilities should do with respect to their own staffing characteristics. To most effectively improve their quality of care, nursing home decision makers are often unable to scale down research findings for their own use.

Often facilities face considerable logistical challenges to changes in staffing (Wunderlich, Sloan, & Davis, 1996). First, they are often not able to make the most desirable staffing changes. For example, eliminating all agency staff. They are more often only able to make incremental positive changes in their staff characteristics. For example, reducing agency staff use by 10%. Second, facilities are often not able to make desirable staffing changes in all areas simultaneously. For example, a facility may have a limited additional specified amount of resources that could be dedicated to changing staffing. Information on what they might do that would be most beneficial would be useful. Research has shown that staffing characteristic relationships with quality are often nonlinear, have interactions, and vary for different caregivers (Castle & Engberg, 2007; Kim, Harrington, et al., 2009). As such, the most efficacious changes in staffing are not always self-evident and depend upon the current position of the facility with respect to their current make-up of staffing characteristics. Of practical use would be a robust, low cost, and easily available tool showing the varying changes in quality a facility could expect from any incremental changes in their current staffing they may choose to implement.

This project provides this guidance via the Internet with a free tool in which any nursing home decision maker or policy maker can determine from a change in each of four staffing characteristics (i.e., staffing levels, turnover, stability, and agency use) the potential yield on quality from such a change. Specifically, we use staffing characteristics and their quality implications for nurse aides (NAs), licensed practical nurses (LPNs), and registered nurses (RNs), with data coming from an existing large (N = 2,946) nursing home primary data collection initiative (i.e., these data are used in a background simulation application for the web-tool).

From entering data on a nursing homes current staffing make-up, the web-tool we developed indicates the potential changes in quality that can be expected from an incremental improvement in any one of the four staffing characteristics. As such, this web-tool bridges the gap between research findings and nursing home practice.

Methods

Data

Four sources of information are used in the web-tool. A Staffing Survey, the On-line Survey Certification and Recording system (OSCAR), and Nursing Home Compare data are used in a simulation analysis and are described below. Additional information on their own staffing characteristics is provided by users when they visit the website (www.crhc.pitt.edu/StaffAssist/).

Staffing Survey.—

As a result of previous survey experience (Castle & Engberg, 2007), we designed a survey to collect information on multiple staffing characteristics of nursing homes. The content of the survey built on what was learned from similar staffing surveys and used validated items (Castle & Engberg, 2007). The survey explicitly asked for data on staffing, turnover, stability, and agency staff on a quarterly basis for RNs, LPNs, and NAs for 2007 and 2008 (these items can be viewed at www.hpm.pitt.edu).

We created our sample by using information from the OSCAR data (described below), and we retained the OSCAR facility ID numbers, so that the primary data could be subsequently merged with secondary data sources. Small nursing homes (<30 beds) and hospital-based facilities were excluded from the sample. A total of 2,946 surveys were returned (response rate = 74%).

OSCAR Data.—

OSCAR contains facility and aggregated resident data routinely collected through the nursing home certification process. The OSCAR is conducted by state licensure and certification agencies. OSCAR data include general facility characteristics including ownership, number of beds, and the average census. Approximately 18,000 facilities are included in the OSCAR (Kash, Hawes, & Phillips, 2007) and all of the facilities included in the primary data collection.

Nursing Home Compare.—

In November of 2002, the Centers for Medicare & Medicaid Services began publicly reporting the quality of almost all nursing homes in the Untied States on this website (www.Medicare.gov/NHCompare). The website has remained available to any user and has updated information on nursing home quality. The quality information provided is primarily presented in a series of quality measures (QMs; Abt Associates, 2003). Eleven long-stay QMs were used in this research (see Table 6).

Statistical Analyses

Simulation models use statistical estimation so that the likelihood of various outcomes can be more accurately estimated. A simulation model was developed using the secondary data (described above). Users of the website enter their individual staffing information (inputs) and the statistical estimation based on this secondary data produces estimates of changes in quality that occur (output). The simulation analysis works as a background application. For practitioners, a user-friendly web-based tool is observed and not estimation equations.

The simulation model was developed from what is currently proposed in the research literature as important considerations in assessing the quality–staffing relationship. Specifically, the specification developed and used by (Castle & Engberg, 2007) was used. This specification includes a multivariate model for each outcome with all four staffing characteristics of interest (including their nonlinear relationships and interactions), seven facility characteristics (e.g., number of beds), and two market characteristics (e.g., competition). More details of the simulation model are available from the authors.

Development of Web-Based Tool

We purposefully developed a website interface that was extremely simple to use and required little investment in time by the user. This was important, as decision aids may have declining impact if information is not provided within 20 min of use (Joseph-Williams et al., 2010). First, the user enters the address of the nursing home. From this address, several fields are populated that are used in the computations. These populated fields include ownership, bed size, and chain affiliation status of the nursing home. This data comes from the OSCAR. A further prompt asks the user if these are correct and to change any incorrect values. This completes the information on the first page of the web-application.

On the second page of the application, a prompt asks the user to complete their own current staffing, turnover, agency use, and stability information. Drop-downs are available for helping calculate these values (if this help is needed). Thus, information is provided by the user on their current facility. Using this information provided and the additional information in the primary database (described above), simulation analyses are performed that predict changes in the QMs, based on unit increases in the staffing characteristics provided by the user.

The following web pages then show the impact on each of the QMs of the unit change in each of the staffing characteristics. This information is provided in a graphical format (i.e., series of stars) indicating the potential level of improvement that can be achieved. For example, the potential level of improvement that can be achieved by decreasing RN turnover by 10% or decreasing NA agency staff use by 10% is shown.

Pilot Testing and Analyses

A focus group was convened, to allow subjects to use the alpha-version of the web-tool and solicit their comments and suggestions on content, ease of use, comprehensibility, and over-all reactions. This focus group consisted of 10 NHAs. We discussed what paths they were looking for that were not present; what words they were looking for; whether the user directions were clear; and if the vocabulary was appropriate.

We also observed these participants using the alpha-version of the web-tool to determine what was and was not working well in the interface. Specifically, we observed participants to see if they completed tasks successfully; how fast they could complete each task; any problems they had; and where they got confused.

The alpha-version of the web-tool was then further shared with 10 experts on staffing/top management/nursing homes. We discussed any additional changes they would recommend to the web-tool. The changes were incorporated into the beta-version of the web-tool.

We conducted a survey of NHAs to determine the usability of the beta-version of the web-tool. This was conducted following the approach of Gehrke and Turban (1999). That is, these authors have provided guidance in the form of questionnaire items that are associated with successful website designs. An invitation was sent by email to 115 NHAs, asking if they would use our web-tool and complete the brief questionnaire. They were randomly chosen from a list of approximately 3,000 NHAs emails maintained by the authors. One hundred NHAs responded, giving a response rate of 87%.

For the nursing homes entering staffing information (i.e., using the web-tool), a brief survey was conducted to determine the utility of the information presented. That is, we asked if the information was useful, if the recommendations made sense, and if the information was used in subsequent staffing decisions.

To assess potential effectiveness, we monitored use of the web-tool after disseminating information of its availability. This included the number of website users (i.e., hits), number of comments posted on the website, and number of emails received by the developers (the email address of the developers was included in the dissemination materials, discussed below).

Descriptive characteristics of the web-tool users were also examined. This included facility characteristics (e.g., bed size) and quality characteristics (e.g., QM scores). Given the format of the web-tool, we were able to stratify these descriptive characteristics by nursing homes simply visiting the website and those entering staffing information (i.e., more active users).

In addition, as a further guide to assess the potential effectiveness of the web-tool, we examined changes in the QMs scores for the nursing homes identifying themselves as using the information/recommendations provided by the web-tool. The scores for the 11 QMs 6 months after use of the web-tool were compared with the scores at the time the web-tool was first used.

Results

The focus group of 10 NHAs provided more than 40 comments on the alpha-version of the web-tool with respect to content and ease of use. Ten example comments are provided in Table 1. The alpha-version of the web-tool was also shared with 10 experts on staffing/top management/nursing homes. Thirty-five comments with respect to content and ease of use were provided. Ten example comments are shown in Table 1.

Table 1.

Modifications to Alpha-Version of Website

Nursing home administrators feedback (N = 10) 
    1. Move boxes to center of page, creating less “blank” space. 
    2. Key at bottom to interpret stars and colors. 
    3. Include STEPS (e.g., STEP 1, STEP 2, STEP 3). 
    4. Have a next step prompt, but need a back button also. 
    5. Provide a key for the green stars and what it means for improvements. 
    6. For STEP 3, put these as sections A, B, C, and D on different pages. 
    7. Statement on first page that will not be used for SPAM. 
    8. Statement on first page, that no additional emails will result if the website is used. 
    9. In step 1, use Provider ID Number (NPI#). 
    10. Included a section on the first page labeled PRIVACY POLICY. 
Content expert feedback (N = 10) 
 1. Layering of information would be beneficial.a 
    2. Include additional resources sections. 
    3. Include best practices section. 
    4. Examine color and font style for optimal understanding. 
    5. Do not use “pop-ups.” 
    6. Font size may not be critical, but use largest size that fits on a page. 
    7. Nursing Home Compare technical reports may help in how to present output. 
    8. Remove “Our Advice.” 
    9. Address what happens if simulation shows a decrease in quality, rather than improvement in quality. 
    10. Address what happens if some of the information is not provided. 
Nursing home administrators feedback (N = 10) 
    1. Move boxes to center of page, creating less “blank” space. 
    2. Key at bottom to interpret stars and colors. 
    3. Include STEPS (e.g., STEP 1, STEP 2, STEP 3). 
    4. Have a next step prompt, but need a back button also. 
    5. Provide a key for the green stars and what it means for improvements. 
    6. For STEP 3, put these as sections A, B, C, and D on different pages. 
    7. Statement on first page that will not be used for SPAM. 
    8. Statement on first page, that no additional emails will result if the website is used. 
    9. In step 1, use Provider ID Number (NPI#). 
    10. Included a section on the first page labeled PRIVACY POLICY. 
Content expert feedback (N = 10) 
 1. Layering of information would be beneficial.a 
    2. Include additional resources sections. 
    3. Include best practices section. 
    4. Examine color and font style for optimal understanding. 
    5. Do not use “pop-ups.” 
    6. Font size may not be critical, but use largest size that fits on a page. 
    7. Nursing Home Compare technical reports may help in how to present output. 
    8. Remove “Our Advice.” 
    9. Address what happens if simulation shows a decrease in quality, rather than improvement in quality. 
    10. Address what happens if some of the information is not provided. 
a

That is, additional information that can be identified via the web-tool. We include layering information on nursing home compare, staffing regulations, science behind the research, and our research.

Table 2 presents the findings from the survey conducted with 100 NHAs to determine the usability of the beta-version of the web-tool. For all sections, the NHAs’ reactions were very positive. For example, 92% agreed or strongly agreed that the website was “easy to use.”

Table 2.

Evaluation of Beta-Version of Website

Question Percent 
Overall use Agree or strongly agree 
    I think I will use this website frequently 87 
    I found the website easy to use 96 
    Task took a reasonable amount of time to complete 100 
    I found the functions of the website well integrated 98 
    Most people would be able to use this website 95 
    I felt confident in using this website 90 
    I thought the results were useful 81 
    I thought the results were believable 85 
Web pages Easy or very easy 
    Reading characters on the page 97 
    Organization of information 92 
    Sequence of pages 100 
 Agree or strongly agree 
    Website is visually appealing 100 
    Individual pages are well designed 87 
Learning Easy or very easy 
    Learning to use the website 89 
    Exploring the features of the website 92 
    Performing tasks on the website 94 
    Supplemental materials provided 95 
Terminology and website information Consistent or very consistent 
 Use of terms throughout the website 100 
    Intuitiveness of terms used 95 
    Position of messages on the screen 92 
    Prompts for input 89 
    Website informs about progress 84 
Website capabilities Consistent or very consistent 
    Website speed 87 
    Website reliability 89 
    Correcting mistakes 85 
Overall reaction to website Agree or strongly agree 
    Overall, I am satisfied with how easy it was to use 92 
    Overall, this website has the functions I expect 86 
    Overall, this website has the capabilities I expect 87 
    Overall, I am satisfied with this website 92 
Question Percent 
Overall use Agree or strongly agree 
    I think I will use this website frequently 87 
    I found the website easy to use 96 
    Task took a reasonable amount of time to complete 100 
    I found the functions of the website well integrated 98 
    Most people would be able to use this website 95 
    I felt confident in using this website 90 
    I thought the results were useful 81 
    I thought the results were believable 85 
Web pages Easy or very easy 
    Reading characters on the page 97 
    Organization of information 92 
    Sequence of pages 100 
 Agree or strongly agree 
    Website is visually appealing 100 
    Individual pages are well designed 87 
Learning Easy or very easy 
    Learning to use the website 89 
    Exploring the features of the website 92 
    Performing tasks on the website 94 
    Supplemental materials provided 95 
Terminology and website information Consistent or very consistent 
 Use of terms throughout the website 100 
    Intuitiveness of terms used 95 
    Position of messages on the screen 92 
    Prompts for input 89 
    Website informs about progress 84 
Website capabilities Consistent or very consistent 
    Website speed 87 
    Website reliability 89 
    Correcting mistakes 85 
Overall reaction to website Agree or strongly agree 
    Overall, I am satisfied with how easy it was to use 92 
    Overall, this website has the functions I expect 86 
    Overall, this website has the capabilities I expect 87 
    Overall, I am satisfied with this website 92 

Note:N = 100 nursing home administrators. Information collected via an anonymous mail survey.

Table 3 presents some basic statistics regarding use of the website. For example, 6 months after the website was fully operational, it had received 2,233 visits (i.e., hits). A total of 211 positive comments were received, including comments such as “much-needed resource” “interesting use of the web,” and “will use frequently.”

Table 3.

Evaluation of Website Use

Characteristic Number Evaluation metrics (characteristic/hits), % 
Visits to website (i.e., hits) 2,233 — 
Staffing data entered 450 20.15 
Posted “practices used” 241 10.79 
Provided website comments 
    Positive 211 9.45 
    Negative 45 2.01 
Provided additional suggestions % to improve website 32 1.43 
Characteristic Number Evaluation metrics (characteristic/hits), % 
Visits to website (i.e., hits) 2,233 — 
Staffing data entered 450 20.15 
Posted “practices used” 241 10.79 
Provided website comments 
    Positive 211 9.45 
    Negative 45 2.01 
Provided additional suggestions % to improve website 32 1.43 

Many nursing home decision makers provided comments on how information was used from this web-tool. These comments were provided both in the comments section of the website and as email communications to the developers. Example uses of the information include presentations to board members, staff, and use in mailings to family members.

We were also able to characterize nursing homes visiting the website (provided in Table 4). This shows that facilities visiting the website and providing staffing characteristic information were similar, with respect to factors such as size and ownership, to those in the primary data from 2,946 nursing homes. However, facilities visiting the website and providing staffing characteristic information were more likely to be of lower quality (as measured by the QMs) and have less favorable staffing.

Table 4.

Characteristics of Nursing Homes Using Website

Characteristic Visited website (N = 2,233) Entered staffing information (N = 450) 
Staffing characteristics 
 Staffing levels (FTEs per 100 residents) 
        RNs — 11.3 
        LPNs — 14.3 
        NAs — 28.8* 
    Turnover (% in past 3 months) 
        RNs — 18.6* 
        LPNs — 21.2 
        NAs — 23.1 
Agency staff use (% positions in past 3 months) 
        RNs — 8.2 
        LPNs — 9.8 
        NAs — 16.2 
Quality characteristicsa (%) 
    Percent physical restraint use 10.8 14.2 
    Percent with moderate-to-severe pain 5.3 5.9 
    Percent residents with catheter in bladder 8.8 9.4 
    Percent low-risk residents with pressure sores 7.9 7.7 
Facility characteristics 
    For-profit (%) 54 52 
    Chain member (%) 46 48 
    Organizational size (# beds) 142 139 
    Occupancy rate (%) 84* 83* 
    Medicaid occupancy rate (%) 58* 56* 
Characteristic Visited website (N = 2,233) Entered staffing information (N = 450) 
Staffing characteristics 
 Staffing levels (FTEs per 100 residents) 
        RNs — 11.3 
        LPNs — 14.3 
        NAs — 28.8* 
    Turnover (% in past 3 months) 
        RNs — 18.6* 
        LPNs — 21.2 
        NAs — 23.1 
Agency staff use (% positions in past 3 months) 
        RNs — 8.2 
        LPNs — 9.8 
        NAs — 16.2 
Quality characteristicsa (%) 
    Percent physical restraint use 10.8 14.2 
    Percent with moderate-to-severe pain 5.3 5.9 
    Percent residents with catheter in bladder 8.8 9.4 
    Percent low-risk residents with pressure sores 7.9 7.7 
Facility characteristics 
    For-profit (%) 54 52 
    Chain member (%) 46 48 
    Organizational size (# beds) 142 139 
    Occupancy rate (%) 84* 83* 
    Medicaid occupancy rate (%) 58* 56* 

Notes: FTE = full-time equivalent; LPN = licensed practical nurse; NA = nurse aide; RN = registered nurse.

a

Four quality measures (of 10 used on website) reported for parsimony.*Significantly different at the p = .05 level from the 2,946 nursing homes in primary data.

The 450 nursing homes entering staffing information were surveyed regarding the utility of the information given by the web-tool. A total of 347 responses were received (response rate = 77%). These results are provided in Table 5. Respondents indicated that for the most part, some of the information provided as part of the web-tool was used (i.e., 69% of respondents).

Table 5.

Utility of Information Reported by Respondents Using Website

Characteristic Percent (n) or M (SDRange 
Use of web-tool 
    Estimated time used to enter staffing information (min) 2.4 (1.3) 1–12 
    Estimated total time using web-tool (min) 13.4 (5.7) 1–48 
    Use of other web-links (yes) 78% (78) — 
        Number web-links used 4.1 (1.3) 1–9 
    Use of other resources (yes) 74% (74) — 
        Number of other resources used 2.2 (0.8) 1–4 
Overall use of web-tool information 
    Used staffing information in some way 69% (69) — 
    Used staffing information in an extensive way 23% (23) — 
    Used staffing information in some operational decision making 77% (77) — 
    Used staffing information in extensive operational decision making 42% (42) — 
Specific use of web-tool information 
    Staffing level recommendations from web-tool were followed 9% (9) — 
    Staffing level recommendations were used in some way 11% (11) — 
    Turnover recommendations from web-tool were followed 12% (12) — 
    Turnover recommendations were used in some way 14% (14) — 
    Agency staff recommendations from web-tool were followed 18% (18) — 
    Agency staff recommendations were used in some way 21% (21) — 
    Staff stability recommendations from web-tool were followed 8% (8) — 
    Staff stability recommendations were used in some way 9% (9) — 
Characteristic Percent (n) or M (SDRange 
Use of web-tool 
    Estimated time used to enter staffing information (min) 2.4 (1.3) 1–12 
    Estimated total time using web-tool (min) 13.4 (5.7) 1–48 
    Use of other web-links (yes) 78% (78) — 
        Number web-links used 4.1 (1.3) 1–9 
    Use of other resources (yes) 74% (74) — 
        Number of other resources used 2.2 (0.8) 1–4 
Overall use of web-tool information 
    Used staffing information in some way 69% (69) — 
    Used staffing information in an extensive way 23% (23) — 
    Used staffing information in some operational decision making 77% (77) — 
    Used staffing information in extensive operational decision making 42% (42) — 
Specific use of web-tool information 
    Staffing level recommendations from web-tool were followed 9% (9) — 
    Staffing level recommendations were used in some way 11% (11) — 
    Turnover recommendations from web-tool were followed 12% (12) — 
    Turnover recommendations were used in some way 14% (14) — 
    Agency staff recommendations from web-tool were followed 18% (18) — 
    Agency staff recommendations were used in some way 21% (21) — 
    Staff stability recommendations from web-tool were followed 8% (8) — 
    Staff stability recommendations were used in some way 9% (9) — 

Note:N = 100 web-tool users. Information collected via a web-survey.

Of the 450 nursing homes entering staffing information, 213 had QM information 6 months post usage. We were able to determine that for these facilities, an average of 5 of the 11 QMs improved by more than 1% (shown in Table 6).

Table 6.

Changes in Quality Measures of Nursing Homes Using Web-Tool (N = 213)

Quality measure Baseline, percent (SDSix-month follow-up, percent (SDPercent change (baseline minus follow-up) 
Percent of residents whose need for help with daily activities has increased 18.0 (8.3) 15.8 (8.8) 2.2* 
Percent of residents who have moderate-to-severe pain 5.1 (4.3) 4.7 (4.7) 0.4 
Percent of high-risk residents who have pressure sores 15.5 (6.8) 12.5 (7.0) 
Percent of low-risk residents who have pressure sores 2.7 (3.4) 2.3 (2.8) 0.4 
Percent of residents who were physically restrained 7.0 (6.9) 5.9 (7.2) 1.1 
Percent of residents who are more depressed or anxious 13.4 (9.6) 14.1 (9.2) 0.7 
Percent of low-risk residents who lose control of their bowels or bladder 48.6 (15.0) 46.2 (13.0) 2.4 
Percent of residents who have/had a catheter inserted and left in their bladder 5.8 (4.1) 5.7 (4.0) 0.1 
Percent of residents who spent most of their time in bed or in a chair 6.1 (5.6) 4.3 (5.6) 1.8 
Percent of residents whose ability to move about in and around their room got worse 11.9 (7.4) 12.6 (7.5) 0.7 
Percent of residents with a urinary tract infection 8.9 (5.2) 8.9 (5.4) 0.0 
Percent of residents who lose too much weight 10.3 (5.4) 8.4 (5.0) 1.9 
Quality measure Baseline, percent (SDSix-month follow-up, percent (SDPercent change (baseline minus follow-up) 
Percent of residents whose need for help with daily activities has increased 18.0 (8.3) 15.8 (8.8) 2.2* 
Percent of residents who have moderate-to-severe pain 5.1 (4.3) 4.7 (4.7) 0.4 
Percent of high-risk residents who have pressure sores 15.5 (6.8) 12.5 (7.0) 
Percent of low-risk residents who have pressure sores 2.7 (3.4) 2.3 (2.8) 0.4 
Percent of residents who were physically restrained 7.0 (6.9) 5.9 (7.2) 1.1 
Percent of residents who are more depressed or anxious 13.4 (9.6) 14.1 (9.2) 0.7 
Percent of low-risk residents who lose control of their bowels or bladder 48.6 (15.0) 46.2 (13.0) 2.4 
Percent of residents who have/had a catheter inserted and left in their bladder 5.8 (4.1) 5.7 (4.0) 0.1 
Percent of residents who spent most of their time in bed or in a chair 6.1 (5.6) 4.3 (5.6) 1.8 
Percent of residents whose ability to move about in and around their room got worse 11.9 (7.4) 12.6 (7.5) 0.7 
Percent of residents with a urinary tract infection 8.9 (5.2) 8.9 (5.4) 0.0 
Percent of residents who lose too much weight 10.3 (5.4) 8.4 (5.0) 1.9 

*Significantly different at the p = .05 level.

Discussion

A substantial number of users visited the web-tool (N = 2,233). Of these users, 450 entered staffing information. With no benchmark information, we have no way of evaluating whether this number of users represents a successful web-tool application or not. Still, given the number of posted comments and article downloads, these users would seem to be benefiting from the information provided. Moreover, as discussed below, web-tool users did show significant improvements in some of their QMs over time.

That so many web-tool users downloaded the reports and summaries was somewhat surprising. Many of the statistics provided are available on professional association websites (such as the American College of Health Care Administrators [ACHCA]). We speculate that the reports and summaries were used as an adjuvant to the types of information provided by groups such as ACHCA and were seen as less biased. We make this speculation based on the stated uses of our website by users which were, in many cases, for board presentations and dissemination of information to other parties, such as staff and consumers. Thus, for some purposes, the information we provided may be viewed as less partisan than from professional associations.

One added value of the web-tool was the opportunity for users to provide accounts of their staffing changes and “fixes.” Users seemed very willing to share problems and solutions with others. They were also willing to share comments to improve the web-tool. One particularly interesting suggestion was to have an export data function, so that data could be placed in Excel or Access, for subsequent use by decision makers.

Based on the apparent willingness of some NHAs to share information in this way, we speculate that an electronic mentoring (e-mentoring) program or a like-communities initiative may be well received and useful for further helping some NHAs improve their facilities’ quality. E-mentoring facilitates mentor interactions via the web (see Cothran et al., 2009). Like communities place users in a similar status together (e.g., high staff turnover) and uses the platform to share experiences (see for example PatientsLikeMe; Wicks et al., 2010).

Further suggestions provided by web-tool users were to include market-level information. This involved providing benchmarks and comparisons with facilities in the county or state. Clearly, NHAs are familiar with the provision of information in this way as part of the functions of Nursing Home Compare. Our data were not sufficiently large to allow such comparisons. In addition, the web-tool was not developed with this function in mind. Still, these suggestions may be indicative of the wide range of sophistication in data and information management that exists in nursing home top management. Many facilities are known to be technologically poor, but some management would appear to be very tech-savvy.

For the less tech-savvy, a “detailing” function in this area as part of professional education or licensure may be of use. That is, motivational interviewing (Miller & Rollnick, 2002) or more tailored health communication (Enwald & Huotari, 2010) may be effective interventions to help nursing home decision makers improve facility staffing characteristics.

We were able to determine that, for these facilities, an average of 5 of the 11 QMs improved by more than 1%. It may be unrealistic to expect significant improvements in all of the QMs over the time period examined. As Castle and Engberg (2008) described, some QMs may be more sensitive to staffing characteristics than others. They propose that physical restraint use, catheter use, pain management, and pressure sores may be more staffing sensitive than other QMs (Castle & Engberg, 2008). We find significant improvements in only two of these QMs. This potentially weakens our findings, but we note that our knowledge on which QMs are staffing sensitive is underdeveloped.

The combined impact of these changes in QMs likely has practical significance on resident care. If these resident care improvements entail less clinical interventions, less overall staff time to care for these residents, and the improved quality attracts more potential private-pay residents to the facility; then, this may be useful in making the business case that staffing improvements are beneficial to the facility. However, this must be interpreted with trepidation, as we have no way of knowing whether these changes were due to staffing changes or one of many other changes that could influence quality of care.

That nursing homes with the lowest quality entered staffing information could be interpreted to mean that the web-tool has most utility for these facilities. This information could be used to more effectively target facilities to use the web-tool. However, nursing homes with more favorable QM scores may have used the website for purposes other than making staff changes (such as dissemination of the reports or use of the additional web-links). Moreover, most of these facilities with better QM scores could likely further improve quality through improvements in staffing characteristics.

From the 450 nursing homes entering staffing information, a brief survey was conducted to determine the utility of the information presented. Respondents indicated that the distinctions between agency use, stability, and turnover were useful, and for the most part, the information provided as part of the web-tool was followed (especially for agency staff use). However, staffing levels in most cases trumped these other possible staffing changes. That is, if the option existed to replace a current caregiver, this was almost always followed (irrespective of the web-tool predictions). The distinction in use of the web-tool seemed to rest on whether the facility had some new or additional resources to dedicate to staffing changes or whether a staff member had to be replaced.

A formal cost-benefit evaluation of the web-tool was not conducted. The data used to develop the web-tool were collected as part of a prior research project. Thus, arguably the most-expensive cost was not incurred and the web-tool primarily consisted of website development and testing. The web-based resource may be a feasible low-cost model that can be replicated for other important areas of nursing home care. This could include areas in which other secondary sources of data are readily available, for example, satisfaction information, best practices, resident safety, and incident reporting.

The apparent success of our web-tool may be indicative of the need for more tailored information for nursing home decision makers and/or more consideration to the information communication technology used. For example, the repository of knowledge for staffing could be placed in a wiki application (i.e., an interactive database; see Bastida, McGrath, & Maude, 2010). Or more simply, findings may reflect that NHAs “surf the web” and use multiple websites.

Limitations and Suggestions for Future Modifications

One limitation of the web-tool is that more specific unit-level information may be more useful for nursing home decision makers. The web-tool reports aggregate changes in quality, whereas changes at a unit level may be more useful. Such refinements may be possible in the future by using the Minimum Data Set (MDS) information. That is, a “drill down” format for the information presented would be beneficial.

As noted above, the inability to provide a robust evaluation of quality improvements coming from using the web-tool is a further limitation. Many changes (other than staffing characteristics) could influence quality of care. The findings reported may also be the result of regression to the mean. Moreover, it would be advantageous to determine the accuracy of the website predictions. We note that the information provided by new users is added to the data used for the simulation analyses potentially increasing the precision of the calculated changes in staffing characteristics. In this way, the predictions should become more robust over time. However, with the recent release of the MDS 3.0, we may have to revisit and potentially alter the simulation analyses in the future. It is unclear as to the impact the MDS 3.0 will have on the QMs and our simulation predictions.

We also note that the web-tool facilitates “action” in the area of staffing. It would be heartening to find the predictions to be reliable and accurate. However, the possibility exists that the presence of the web-tool facilitates changes, which on the whole benefits resident care.

Conclusions

We are clearly in an e-centric era. With e-Health, e-bay, e-Invite, e-pharmacy, e-physician, a multitude of “e” web resources exist to help. An e-Manager resource for NHAs may be a resource to consider in the future. The evidence collected as part of implementing this research-to-practice web-tool would suggests that this may be a much-needed resource. The web-tool we developed was useful in informing nursing home top management of the importance of staffing and facilitating staffing changes. The web-based resource may be a feasible low-cost model that can be replicated for other quality improvement tools in nursing homes. The web-tool is available at www.crhc.pitt.edu/StaffAssist.

References

Abt Associates
Validation of long-term and post-acute quality indicators
 , 
2003
Cambridge, MA
Author
Bastida
R
McGrath
I
Maude
P
Wiki use in mental health practice: Recognizing potential use of collaborative technology
International Journal of Mental Health Nursing
 , 
2010
, vol. 
19
 (pg. 
142
-
148
)
Castle
NG
Nursing home caregiver staffing levels and quality of care: A literature review
Journal of Applied Gerontology
 , 
2008
, vol. 
27
 (pg. 
375
-
405
)
Castle
NG
Engberg
J
The influence of staffing factors on quality of care in nursing homes
Health Services Research
 , 
2007
, vol. 
42
 (pg. 
1822
-
1847
)
Castle
NG
Engberg
J
Further examination of the influence of caregiver staffing levels on nursing home quality
The Gerontologist
 , 
2008
, vol. 
48
 (pg. 
464
-
476
)
Cothran
D
McCaughtry
N
Faust
R
Garn
A
Kulinna
PH
Martin
J
E-mentoring in physical education: Promises and pitfalls
Research Quarterly for Exercise & Sport
 , 
2009
, vol. 
80
 (pg. 
552
-
562
)
Enwald
HP
Huotari
ML
Preventing the obesity epidemic by second generation tailored health communication: An interdisciplinary review
Journal of Medical Internet Research
 , 
2010
, vol. 
12
 pg. 
e24
 
Gehrke
D
Turban
E
Determinants of successful website design: Relative importance and recommendations for effectiveness. Proceedings of the 32nd Hawaii International Conference on System Science
 , 
1999
Maui, Hawaii
Hyer
K
Temple
A
Johnson
CE
Florida's efforts to improve quality of nursing home care through nurse staffing standards, regulation, and Medicaid reimbursement
Journal of Aging and Social Policy
 , 
2009
, vol. 
21
 (pg. 
318
-
337
)
Joseph-Williams
N
Evans
R
Edwards
A
Newcombe
RG
Wright
P
Grol
EG
Supporting informed decision making online in 20 minutes: An observational web-log study of a PSA test decision aid
Journal of Medical Internet Research
 , 
2010
, vol. 
12
 
2
pg. 
e15
 
Kash
BA
Hawes
K
Phillips
CD
Comparing staffing levels in the online survey certification and reporting (OSCAR) system with the Medicaid cost report data: Are differences systematic?
The Gerontologist
 , 
2007
, vol. 
47
 (pg. 
480
-
489
)
Kim
H
Harrington
C
Greene
W
Registered nurse staff mix and nursing home quality
The Gerontologist
 , 
2009
, vol. 
49
 (pg. 
81
-
90
)
Kim
H
Kovner
C
Harrington
C
Greene
W
Mezey
M
A panel data analysis of the relationships of nursing home staffing levels and standards to regulatory deficiencies
Journal of Gerontology: Social Sciences
 , 
2009
, vol. 
64B
 (pg. 
269
-
278
)
Miller
WR
Rollnick
S
Motivational interviewing, second edition: Preparing people for change
 , 
2002
New York
The Guilford Press
Wicks
P
Massagali
M
Frost
J
Brownstein
C
Okun
S
Vaughan
T
, et al.  . 
Sharing health data for better outcomes on PatientsLikeMe
Journal of Medical Internet Research
 , 
2010
, vol. 
12
 pg. 
e19
 
Wunderlich
GS
Sloan
FA
Davis
CK
Nursing staff in hospitals and nursing homes: Is it adequate?
 , 
1996
Washington, DC
Committee on the Adequacy of Nurse Staffing in Hospitals and Nursing Homes, Division of Health Care Services, Institute of Medicine, National Academy Press

Author notes

Decision Editor: Kathleen Walsh Piercy, PhD