The Seniors’ Outdoor Survey: An Observational Tool for Assessing Outdoor Environments at Long-Term Care Settings

Abstract

Purpose of the Study:

To describe the development and psychometric testing of the Seniors’ Outdoor Survey (SOS), an instrument for evaluating how well the outdoor space in a long-term care setting supports the preferences and outdoor usage of residents.

Design and Methods:

Content validity of the main SOS items initially was based on relevant literature and preliminary studies in diverse long-term care settings. After conducting a multiregional pilot study with 152 outdoor spaces at 68 assisted living facilities, the instrument was substantially revised and tested for interrater and test–retest reliability with 22 outdoor spaces at 12 long-term care settings, using 2 raters. Validity was examined using content analysis of resident survey responses (N = 1,128) from the multiregional study and specific item validation by subject matter experts (N = 53).

Results:

The final instrument contains 60 ratable items organized in 5 domains: access to nature (14 items), outdoor comfort and safety (15 items), walking and outdoor activities (14 items), indoor–outdoor connection (11 items), and connection to the world (6 items). Intraclass correlation coefficient (ICC) estimates of interrater reliability were .91 for the overall instrument, ranging from .83 to .98 for the 5 domains. Interrater reliability (ICC) was above .70 for more than 79% of individual items. Test–retest reliability (ICC) was .92, ranging from .81 to .98 for domains.

Implications:

The SOS tool fills a gap in the available environmental assessment instruments, providing a reliable way for researchers, providers, and designers to evaluate and compare the supportive potential of outdoor spaces for long-term care residents.

Environmental assessment instrument, Validation, Reliability, Outdoor usage, Access to nature, Long-term care residents

Access to outdoor space can provide important health-related benefits for older adults, especially in long-term care settings where residents seldom leave the facility. Spending time outdoors can potentially improve mood, sleeping patterns, and Vitamin D absorption and may reduce falls and fractures (Cohen-Mansfield & Werner, 2001; Detweiler et al., 2012; Joseph, 2006). An increasing number of studies show that outdoor usage, although influenced by weather, health conditions, and staff attitudes, is strongly linked to the characteristics of the physical environment. As age-related disabilities increase in later life, the supportive potential of the physical environment becomes an increasingly important influence on behavior (Cutler, 2000; Lawton, Weisman, Sloane, & Calkins, 1997; Sugiyama & Ward Thompson, 2007). Reported environmental barriers to outdoor usage include inadequate shade and seating, unsafe walkways, and self-locking doors (Access to Nature for Older Adults, 2014; Anderzhon, Fraley, & Green, 2007; Cohen-Mansfield, 2007; Cutler & Kane, 2005; Rodiek, Lee, & Nejati, 2014). Because spending time outdoors is not a required activity, the systematic evaluation of outdoor access remains a low priority, in spite of wide acceptance as a meaningful and health-promoting component of long-term care settings (Hiatt, 1980; Rodiek & Schwarz, 2005, 2007). Without a suitable evaluation instrument, it is difficult to determine which environmental characteristics most effectively support outdoor usage and satisfy residents’ preferences. This paper describes the development of an observational instrument to meet this need.

Existing Environmental Assessment Instruments

Compared with instruments developed to assess physiological, psychological, and cognitive domains, relatively few instruments focus on older adults’ physical environments (Kane, 2000). Several existing instruments (e.g., the Multiphasic Environmental Assessment Procedure [MEAP], by Moos & Lemke, 1996) assess the physical environment as a single component of the overall care setting. Reviews by Cutler (2000) and the Agency for Healthcare Research and Quality (2007) described more than 50 instruments for assessing dementia and nondementia care settings. All instruments were focused on indoor elements, although some instruments included a limited number of outdoor items, such as the Therapeutic Environments Screening Scale (TESS-NH, by Sloane et al., 2002), the Environmental Indices (Cutler, Kane, Degenholtz, Miller, & Grant, 2006), the Observable Indicators of Nursing Home Care Quality Instrument (Rantz et al., 2008), the Residential Care Environmental Assessment (RCEA, by Topo, Kotilainen, & Eloniemi-Sulkava, 2012), and a recent physical environment instrument (Chaudhury & Cooke, 2013) developed to accompany the widely used Dementia Care Mapping (DCM) Tool (Kitwood & Bredin, 1992). An earlier instrument, the Physical and Architectural Features (PAF) checklist (by Moos & Lemke, 1980), had included a wide range of outdoor items, but used a yes/no format, capturing only the presence or absence of an element/feature without evaluating the quality and supportive potential of these features. Of the instruments that included several outdoor items, some were limited to a narrow range of environmental qualities, such as physical accessibility (e.g., the Housing Enabler, by Iwarsson, Nygren, & Slaug, 2005). The only published instrument found to contain a detailed assessment of outdoor space for older adults was the Alzheimer’s Garden Audit Tool (AGAT; Cooper Marcus, 2007), a 74-item observational instrument that has not been subjected to psychometric testing and specifically targets environments for advanced dementia. In “active living” research focused on walking and physical activity (http://activelivingresearch.org/), several instruments have been published on outdoor environments for community-dwelling older adults (e.g., the Senior Walking Environmental Assessment Tool [SWEAT], by Cunningham, Michael, Farquhar, & Lapidus, 2005). However, the items typically refer to neighborhood features such as public streets and sidewalks, with limited relevance for residential care facilities. Overall, outdoor-related issues are fragmented among existing instruments, and none were found that comprehensively assess the supportive potential of outdoor environments at nondementia long-term care settings.

Target Goals for Instrument Development

To meet the need for a validated instrument to evaluate outdoor access for long-term care residents, an environmental audit tool was developed. This paper briefly summarizes how preliminary studies were used to develop a pilot instrument, which was tested in a multiregional study, further examined using surveyed resident preferences and expert opinion, revised into the format presented here, and tested for reliability at 22 outdoor spaces. Although some aspects of the previous study phases have been published in association with other findings, the focus here is on describing the support for content validity, the revision of the pilot instrument, and the reliability testing of the revised instrument. Based on preliminary findings and literature review, the main goals we established were that the instrument should be: (a) Comprehensive—addressing the full spectrum of physical environment issues affecting outdoor usage; (b) Observational—focused on observable physical features, rather than policy or programs; (c) Empirically derived—based on empirical support for items, rather than inference from latent therapeutic goals; (d) User centered—focused on supporting the usage and satisfaction of residents; (e) Multidisciplinary—usable by providers, researchers, design practitioners, and consumer advocates, without specialized expertise, to allow comparison among stakeholders; and (f) Widely applicable—appropriate for a range of residential care settings, to reflect the increased blurring between different levels of care.

Kane (2000, p. 527) discusses the need for environmental assessment that is comprehensive and based on “minimizing subjective ratings and emphasizing specific observations.” She further notes the value of instruments that are useful in multiple settings, entail a reasonable amount of time and effort, and are “reliable in the hands of unspecialized trained assessors.” Because a primary goal of the proposed instrument is to assess physical environments from the user’s perspective, Gibson’s “affordance” concept (Gibson, 1979) formed the conceptual basis for assessing environmental features. Affordances (also used in the RCEA instrument listed earlier) emphasize how well an environmental feature supports (or affords) a targeted behavior, rather than evaluating the feature’s inherent qualities. The affordance approach emphasizes the primary targeted outcome (in this case, support for resident outdoor usage) and is hypothesized to increase reliability by reducing the subjectivity of ratings. The proposed instrument is also focused on assessing environments at the micro (i.e., physical) level, rather than globally at the macro level (i.e., including policy and programmatic issues). This distinction was noted by Calkins and Weisman (1999, pp. 136–137), in describing environmental features as a set of “discrete… independent variables” that are “…objective characteristics of the physical environment.” In this sense, the proposed Seniors’ Outdoor Survey (SOS) tool is similar to the TESS instrument, which was described by Lawton and colleagues (2000) as “…a collection of relatively concrete aspects of nursing-home design, many of which have been suggested as desirable physical features” (p. 36). As an environmental audit, the items in this instrument can be evaluated directly, unlike a psychological scale in which internal consistency within a domain is used to confirm the distinction among domains that are based on latent constructs. Broad applicability is an important goal for the SOS tool, because of the increasing diversity in functional capacity among residents at each level of care, and a trend toward multilevel facilities with shared outdoor spaces. The boundaries between skilled nursing, assisted, and independent living are becoming less distinct as these models respond to evolving regulatory requirements and residents’ preference for aging in place (Golant & Hyde, 2008; Oliva, 2013).

To optimize environmental support for physically and functionally vulnerable residents, the SOS tool is designed to assess environments “through the eyes of a frail resident using a walker or wheelchair” (Supplementary Data). The items in the tool are appropriate for persons with mild to moderate dementia, who constitute a substantial percentage of residents in skilled nursing, assisted living, and even independent living. Because outdoor spaces for those with advanced dementia (living in memory-support units) require highly specific environmental considerations, they should be evaluated using a purpose-designed instrument such as the Alzheimer’s Garden Audit Tool (Cooper Marcus, 2007) or the Environment-Behavior Model for Special Care Units (Zeisel et al., 2003). Recently developed comprehensive dementia-specific design guidelines also have sufficient specificity to assist in evaluating outdoor spaces (Berentsen, Grefsrød, & Eek, 2008; Chalfont & Walker, 2013).

Content Validation of Domains and Items

To investigate residents’ preferences for environmental features related to outdoor usage, preliminary studies were conducted at a range of long-term care settings in southeast Texas (Table 1, Studies 1–4). Features reported as important in focus groups and open-ended written survey questions were categorized using content analysis and further examined with photographic preference surveys. Resident responses regarding environmental barriers and preferred features were fairly consistent across the three facility types; the only significant nongender difference was preference for walkways, which decreased with aging (Rodiek, 2006; Rodiek & Fried, 2005).

Table 1.

Summary of Studies Represented

Study title	Methodology	Facilities in study (N)a	Facility sample frame (N)	Facility participation rate	Facilities randomized	Resident participants (N)	Resident sample frame	Resident response rate	Residents randomized	Mean age	Percent female
Preliminary Study 1	Focus group	1 SN, 2 AL, 2 IL	10	50%	No	76	569	13%	No	77.8	92%
Preliminary Study 2	Survey	1 SN, 1 IL	n/a	n/a	n/a	17	168	10%	No	80.8	100%
Preliminary Study 3a	Focus group	7 AL	34	21%	Yes	108	1,541	7%	Yes	83.6	65%
Preliminary Study 3b	Survey	14 AL	34	41%	Yes	211	1,541	13%	Yes	83.6	65%
Preliminary Study 4	Photo survey	14 AL	34	41%	Yes	133	1,541	9%	Yes	83.9	77%
Multiregional	Survey	68 AL	139	49%	Yes	1,128	5,632	20%	No	83.9	79%
Multiregional	Environmental audit	68 AL	139	49%	Yes	n/a	n/a	n/a	n/a	n/a	n/a
SOS reliability	Environmental audit	4 SN, 5 AL, 3 IL	12	100%	No	n/a	n/a	n/a	n/a	n/a	n/a

Notes: SOS = Seniors’ Outdoor Survey.

^aIL: independent living facilities in the sample were congregate dwellings with communal dining and minimal assistance; AL: assisted living facilities provided assistance with daily activities; and SN: skilled nursing facilities provided assistance with medical needs in addition to daily activities.

Environmental features emerging as important from this process were compared with issues discussed in the relevant literatures, primarily: (a) physical health-related, (b) psychosocial, and (c) design recommendations. Although some overlaps were found, generally the health-related studies addressed environmental features that promote physical activity and nature contact, for benefits such as improved cardiovascular condition, or reduced incidence of falls (e.g., Cohen-Mansfield & Werner, 2001; Detweiler et al., 2012; Joseph, 2006). The psychosocial literature focused on the role of environmental features in supporting therapeutic psychosocial goals (see Regnier [2002], for a review of widely accepted conceptual frameworks). These therapeutic goals, including items such as autonomy, security, and privacy, also appear in the “key dimensions” proposed by Sloane, Zimmerman, and Walsh (2001), based on an earlier model from Lawton, Weisman, Sloane, and Calkins (1997). The design recommendation literature showed consistency among experts and emphasized environmental features reported to influence outdoor usage and satisfaction (e.g., Brawley, 2006; Regnier, 2002). Features prominent in the literature reviews were cross-referenced with those prominent in our preliminary studies to develop the list of environmental features to be included in the pilot instrument; these were categorized into seven domains according to Holsti (1969): (a) safety and security; (b) comfort and accessibility; (c) choice, control, and freedom; (d) access to nature; (e) connection to the world beyond the facility; (f) indoor–outdoor connection; and (g) support for physical activity. Safety and security are linked to underusage of outdoor areas, concern about visual surveillance, and fear of falling due to paving problems (Fonad, Wahlin, Heikkila, & Emami, 2006; Rantakokko et al., 2009). Comfort and accessibly designed features are linked to resident satisfaction and outdoor usage (Brawley, 2006; Cohen & Weisman, 1991; Mooney & Nicell, 1992). Choice, control, and freedom are widely supported in therapeutic conceptual frameworks, contributing to residents’ autonomy (Mather, Nemecek, & Oliver, 1997; Namazi & Johnson, 1992). Access to nature is found to reduce stress and promote psychological well-being in older adults (Barnicle & Midden, 2003; Rodiek, 2002). The frequently reported behavior of residents congregating near the facility entry suggests that connection to the world beyond the facility provides stimulation and fosters continuation of meaningful life roles (Carstens, 1993; Kane, 2004). Problems with the indoor–outdoor connection can cause frustration or injury (Demirbilek & Demirkan, 2004; Rodiek, Lee, & Nejati, 2014). Environments that support physical activity may help improve sleeping patterns, reduce depression, reduce cardiovascular risk factors, and reduce the rate of falls and fractures (Gregg, Pereira, & Caspersen, 2000; Rejeski & Mihalko, 2001; Taylor et al., 2004). Several environmental features were found to be related to multiple domains; for example, a poorly designed or maintained indoor–outdoor connection (e.g., doors that are difficult to open) can simultaneously reduce residents’ safety, access to nature, physical activity, and choice/control/freedom.

Developing and Testing Instrument in Multiregional Study

A pilot instrument (available on request from Access to Nature, www.accesstonature.org) was developed for a multiregional study in which it would be used to compare environmental characteristics with resident outdoor usage and satisfaction. The instrument was pretested with trained and untrained raters (n = 5); after iterative revisions, interrater reliability reached .94 (intraclass correlation coefficient [ICC]) for the overall instrument, with only one domain falling below .70. The finalized pilot instrument included 63 items rated from 1–10 (1 = worst, 10 = best), organized into the seven domains described earlier. Field-testing was conducted as part of a multifaceted National Institute on Aging–funded study in three large U.S. metropolitan regions with different climates (Houston, Chicago, and Seattle), at 68 assisted living facilities randomly sampled within a 2-hr driving diameter of the urban cores (Rodiek & Lee, 2009; Rodiek, Lee, & Nejati, 2014). Outdoor spaces to be evaluated at each facility were selected based on evidence of usage, resulting in 152 spaces total (mean 2.2 per facility). Two raters received approximately 10hr of training at facilities not in the sample, where they practiced rating the supportive potential of environmental features (affordances) from the perspective of a frail elderly resident in a wheelchair or walker. Their ratings were compared with those of the instrument developers, and feasible target goals were discussed for specific features. The same pair of raters independently assessed all facilities in the sample, taking approximately 40min to rate each outdoor space. In this multiregional field-test, interrater reliability was .95 (ICC) for the overall pilot instrument, ranging from .91 to .98 for separate domains.

Revising and Finalizing Instrument

The pilot instrument was revised to confirm the relevance and exhaustiveness of the items and domains. Revisions also aimed to simplify and clarify the format, to increase usability for a wide range of future researchers, facility staff, and design practitioners.

Further Validation

The pilot instrument was revised to reflect the findings of the previously described multiregional study (Table 1), using data from the environmental audits and surveyed resident preferences (Table 2). Considering resident preference as the “gold standard” for quality of life issues (Kane et al., 2004), the open-ended survey responses (n = 1,128) regarding the features they “liked best,” “liked least,” and “would most like to add” were categorized and quantified to examine the validity of items in the pilot instrument. Minor regional variations were found in resident preferences, but no items were excluded on this basis. Validity was also examined by comparing researchers’ and residents’ ratings of selected environmental characteristics at the same facilities; significant agreement was found for three of the five domains tested by this method (Rodiek, 2008). Using an outcomes-based approach, residents’ levels of outdoor usage and walking were compared with the environmental audit ratings of outdoor space at their facilities; multivariate regression analysis found significant positive association with several environmental features (Rodiek & Lee, 2009; Rodiek, Lee, & Nejati, 2014). The validity of domains and items was further examined by 53 subject matter experts selected based on their record of publications on this topic. They rated each item on the revised tool as high, medium, or low, in terms of its potential to support outdoor usage by older adults; Lawshe’s (1975) approach was used to determine content validity and item inclusion (also Wilson, Pan, & Schumsky, 2012).

Table 2.

Validity Support, Distribution, and Reliability of the SOS^a

Support for validityb				Environmental categories/itemsc		Distribution in a sample of 22 outdoor spaces		Reliabilityd
								Interrater ICC	Test–retest ICC
Literature	Preferences	Outcome- based	Expert opinion
						Me	SDe	2 Raters	Rater 1	Rater 2	Mean
Overall instrument						3.43	1.94	0.91	0.93	0.92	0.92
1. Access to nature (14)						3.30	1.62	0.83	0.93	0.89	0.91
H	H	—	H	1a	Abundance of greenery	4.07	1.24	0.58	0.84	0.78	0.81
H	H	M	M	1b	Diverse mix of plants and trees	3.98	1.27	0.64	0.89	0.76	0.82
M	H	—	H	1c	Abundant flowers and color	4.02	1.53	0.91	0.94	0.91	0.92
M	—	H	H	1d	Easily reachable or raised plants	3.55	1.39	0.74	0.95	0.73	0.84
L	—	—	L	1e	Hard boundaries screened by plantsf	—	—	—	—	—	—
M	M	L	H	1f	Seating has pleasant views	3.75	1.40	0.83	0.86	0.96	0.91
L	M	H	M	1g	Water features available	1.66	1.21	0.93	0.89	0.96	0.92
L	—	—	L	1h	Features with movementf	—	—	—	—	—	—
L	L	L	M	1i	Amenities for pets	2.89	1.57	0.86	0.90	0.91	0.91
H	H	H	M	1j	Amenities for birds and wildlife	1.43	0.7	0.62	0.49	0.57	0.53
L	—	H	M	1k	Can see domesticated animalsf	—	—	—	—	—	—
L	L	—	M	1l	Outdoors fairly quiet	3.70	0.98	0.10	0.47	0.74	0.60
M	L	—	M	1m	Privacy from resident rooms	4.00	1.66	0.55	0.9	0.57	0.73
M	L	—	M	1n	Private places to sitf	—	—	—	—	—	—
2. Outdoor comfort and safety (15)g						3.17	1.92	0.94	0.95	0.95	0.95
H	H	L	H	2a	Plenty of seating available	3.43	1.59	0.93	0.96	0.95	0.95
H	L	—	H	2b	Choice of different seating types	2.86	1.89	0.94	0.98	0.97	0.97
H	M	—	H	2c	Some seating easily movable	3.41	1.41	0.86	0.80	0.89	0.84
H	H	L	H	2d	Seats available in sun or shade	3.61	1.51	0.71	0.81	0.77	0.79
M	L	L	H	2e	Seats will not tip over	4.07	1.59	0.95	0.95	0.90	0.93
H	M	L	H	2f	Seating has arms and backs	5.32	2.19	0.94	0.93	0.98	0.95
M	H	—	H	2g	Seats comfortably shaped	3.59	1.33	0.85	0.89	0.95	0.92
L	M	—	H	2h	Seats do not get hot or cold	3.63	1.60	0.86	0.87	0.90	0.89
L	M	—	H	2i	Seats with cushions	1.34	1.08	0.97	0.99	0.85	0.92
L	M	—	M	2j	Swing, glider, rocking chairsh	—	—	0.97	0.92	0.92	0.92
H	M	—	H	2k	Tables for coffee, food, etc.	2.75	1.80	0.93	0.97	0.92	0.94
L	L	—	L	2l	Restroom, drinking fountainf	—	—	—	—	—	—
H	M	—	H	2m	Microclimate control	1.27	0.91	0.72	0.80	n/ai	n/ai
H	L	—	H	2n	Smoking areas well separatedf	—	—	—	—	—	—
H	M	—	H	2o	Outdoor area well maintained	4.60	1.03	0.69	0.75	0.84	0.80
3. Walking and outdoor activities (14)						2.98	1.71	0.88	0.85	0.77	0.81
L	H	L	H	3a	Abundant walkways of different lengths	2.16	1.56	0.75	0.74	0.62	0.68
M	L	H	H	3b	Round trip walkways available	2.70	1.76	0.90	0.92	0.76	0.84
H	H	L	H	3c	Paving level, easy for wheelchairs	4.20	1.03	0.71	0.48	0.61	0.54
H	—	H	H	3d	Paving nonskid and nonglare	4.89	1.21	0.09	0.58	0.48	0.53
L	—	—	M	3e	Handrails along some walkwaysf	—	—	—	—	—	—
L	L	L	H	3f	Walkways partly shaded	3.41	1.46	0.87	0.71	0.67	0.69
L	—	M	H	3g	Interesting views from walkways	3.39	1.48	0.78	0.57	0.52	0.55
H	M	—	H	3h	Frequent seating along walkways	2.57	1.69	0.96	0.89	0.92	0.90
M	L	—	H	3i	Some walkway seating in shade	2.73	1.61	0.91	0.54	0.78	0.66
L	M	M	H	3j	Destinations to walk toward	3.59	1.21	0.78	0.85	0.79	0.82
H	H	—	H	3k	Places for social activitiesf	—	—	—	—	—	—
H	H	L	M	3l	Places for recreation and exercise	1.80	1.24	0.62	0.70	0.57	0.64
M	L	—	M	3m	Play areas for children	1.34	0.96	0.91	0.90	0.68	0.79
M	L	—	M	3n	Place for gardening, horticultural therapyf	—	—	—	—	—	—
4. Indoor–outdoor connection (11)						4.30	1.85	0.84	0.89	0.86	0.87
H	—	—	H	4a	Easily reached from indoor commons	4.95	1.28	0.56	0.73	0.87	0.80
H	L	M	H	4b	Visible from main indoor areas	4.09	1.42	0.63	0.81	0.76	0.79
H	—	M	H	4c	Indoor transition space near doorway	3.66	1.63	0.82	0.77	0.64	0.70
M	—	—	H	4d	Outdoor transition space near doorway	3.20	1.97	0.89	0.94	0.92	0.93
L	L	—	M	4e	Multiple ways to reach outdoor area	2.70	2.26	0.86	0.87	0.89	0.88
H	L	H	H	4f	Doors unlocked during daytime	6.63	1.21	0.80	1.00	n/ai	n/ai
H	H	H	H	4g	Doors open with little effort	4.68	1.34	0.77	0.78	0.76	0.77
L	L	H	H	4h	Doors do not close too quickly	4.55	1.50	0.86	0.85	0.86	0.85
L	H	H	H	4i	Automatic door available, easy to use	5.92	1.30	0.85	0.88	0.71	0.79
M	L	H	H	4j	Can easily cross door threshold	3.52	0.82	0.83	0.49	0.69	0.59
L	—	M	H	4k	Wide paved landing outside doorway	5.02	1.23	0.39	0.60	0.34	0.47
5. Connection to the world (6)						3.59	2.45	0.98	0.97	0.99	0.98
M	M	H	M	5a	Area is located near main entry	3.98	3.05	1.00	1.00	1.00	1.00
L	L	M	L	5b	View of vehicles arriving at facility	4.07	2.89	0.99	0.99	1.00	1.00
L	M	—	M	5c	Views of front-door activities	4.14	2.76	0.98	0.94	0.98	0.96
L	M	L	M	5d	Views of off-site scenery	3.27	1.55	0.95	0.92	0.91	0.91
L	L	M	M	5e	Views of nearby streets or traffic	3.30	2.18	0.94	0.97	0.99	0.98
L	L	L	M	5f	Views of off-site buildings, activities	2.82	1.85	0.91	0.9	0.96	0.93

Notes: Values for overall instrument and main domains are in bold. ICC = intraclass correlation coefficient; SOS = Seniors’ Outdoor Survey.

^aThe SOS instrument can be accessed at: http://www.accesstonature.org/resources.html.

^bHigh, medium, low (H, M, L) support for item validity was estimated from listed resources: Literature shows a comparative estimate of the prevalence of support; preferences reflect quantified open-ended resident responses from multiregional study; outcome-based shows items associated with higher levels of outdoor usage and walking; and expert opinion summarizes their H, M, L ratings of items in the final SOS tool.

^cScoring is based on how well each item supports resident outdoor usage (range is 1–7, with 1 = low, 7 = high).

^dType A ICCs using an absolute agreement definition.

^eMean and SD are the average between means and SDs of two raters.

^fReliability not estimated, as items were added or modified during field-test.

^gExterior fencing was not required for any facility types in this sample.

^hMean and SD not calculated, as item was modified during field-test.

ⁱUnable to compute statistic because one set of responses had no variation.

Instrument Revision

The domain framework was revised to reduce redundancy and allocate the environmental features into more relevant categories. Because the conceptually distinct domains “comfort and accessibility” and “safety and security” had contained many overlapping items (e.g., comfort and safety of seating), they were merged into an environmentally relevant “outdoor comfort and safety” domain, using an approach similar to that described by Moos and Lemke (1980) in revising the PAF checklist. Because the “choice, control, and freedom” domain was comprised mainly items that overlapped with multiple domains (e.g., movable seating involves comfort, safety, and choice), the items in this domain were relocated to environmentally specific domains. Items found to be less relevant in field-testing were removed (e.g., walkway width had low variability, and minimal support in resident preferences). Other items with components that were difficult to assess collectively were separated into individual items (e.g., “comfortable seating” was unbundled into the thermal, cushioning, and ergonomic aspects of seating).

Based on usability testing with researchers and facility staff, the overall SOS format was streamlined; the wording of each item was clarified through iterative editing, making it possible to eliminate detailed subdescriptions. Items related to the same environmental component (e.g., seating) were assembled as clustered questions with multiple ratable items (Supplementary Data). This reduced the feeling of repetition while keeping discrete environmental characteristics separate for analysis. Although the pilot instrument used a 1–10 response scale, the revised version used a 1–7 response scale, based on provider staff feedback during pretesting, and the reported higher utility and reliability of a 7-point scale (Preston & Colman, 2000; Weijters, Cabooter, & Schillewaert, 2010). Another advantage of a 7-point scale is that it permits the use of Likert-type response categories, which are likely to be easier for most nonacademic user groups, including design practitioners and consumer advocates. The resultant ordinal scale is consistent with the objectives of this instrument, in which the level of environmental support can be considered higher or lower, but the intervals are not precisely equal, due to the diverse real-world characteristics of the features being evaluated.

Reliability Testing Revised Instrument

After iterative pretesting to refine the instrument and evaluate its future usability by staff, the reliability of the revised SOS instrument (Supplementary Data) was tested using 22 outdoor spaces at 12 long-term care facilities, constituting an exhaustive sample of congregate facilities within a 30-min driving distance of the research center in southeast Texas (Table 1). Facilities represented a diverse mix of care settings, representing the population of interest for this tool. Sites included three independent, five assisted, and four skilled nursing facilities. The independent living facilities provided congregate living with communal dining and add-on assistance with daily activities. The outdoor spaces at two of the facilities were shared among assisted living and nursing residents. Two students (one from psychology, one from landscape architecture) with no previous expertise on the topic received approximately 6hr of training, similar to that used in the multiregional study, at facilities not in the sample. Outdoor spaces were rated independently over a 2-week period, in dry weather and temperatures in an acceptable comfort range for older adults. Retests were conducted after approximately 7 weeks to reduce memory effects while avoiding the effects of seasonal variation; this time frame is slightly shorter than the 3-month retest interval of indoor spaces used in the TESS-NH. No substantial changes were made to the study’s outdoor spaces between test and retest, with the exception of naturally occurring seasonal variations in plant materials.

Reliability Test Method

The ICC was used to assess the interrater and test–retest reliability of the revised instrument, using a two-way mixed model with absolute agreement. ICC values range from 0 to 1, representing the lowest to highest levels of agreement. Although there is no consensus as to the threshold to determine an acceptable ICC value, .60 to .75 is usually considered an acceptable range of the minimum values (Anastasi, 1988; Portney & Watkins, 1993). All statistical analyses were performed in IBM SPSS Statistics 20.

Reliability

As shown in Table 2, ICC for the overall instrument was .91, ranging from .83 to .98 for separate domains. Test–retest reliability (ICC) for the overall instrument was .92, ranging from .81 to .98 for separate domains. For individual items, interrater ICC ranged from .09 to 1.00, with 79% of items above .70. Two items (microclimate control and locked doors) could not be computed for test–retest reliability due to lack of response variability; both items had absolute agreement of approximately 95%. Two items had very low interrater reliability (.09 for nonskid/nonglare paving, and .10 for outdoor quietness) but had near agreement (within 1 point on a 7-point scale) of 59%. Another low item had .39 interrater ICC for paved landings by doors; this item had near agreement of 45%. Two items had moderately low interrater reliability (.62 for views of wildlife, and .69 for outdoor areas well maintained), but high near agreement (91% and 90%, respectively). Additional items added during instrument development and field-testing, such as “support for gardening” and “smoking areas are well separated,” are shown, but reliability is not reported. Interrater and test–retest reliability were also analyzed for each of the three facility types (assisted living, skilled nursing, and independent living) separately, to further assess the potential variations in the instrument reliability across facility types. Although the results showed some variations, all individual test results were higher than the overall results of .907 (interrater) and .921 (test–retest). Specific findings were: .952 (test–retest) and .930 (interrater) for assisted living facilities (n = 7); .937 (test–retest) and .909 (interrater) for skilled nursing facilities (n = 9); and .954 (test–retest) and .964 (interrater) for independent living facilities (n = 6).

Discussion

Overall, the revised SOS instrument was able to sensitively evaluate a large number of environmental features in a short time period. It exhibited reasonably high reliability, especially in light of the relatively low training requirements for the raters, the subjective nature of the evaluations, the use of a 7-point scale, the mixed range of facility types, and the small number of spaces rated. The overall reliability of .91 found for the SOS instrument compared favorably with that exhibited by the TESS-NH, on which overall interrater agreement was above 80% (Sloane et al., 2002). On comparable instruments, interrater reliability of .75 or higher is considered good (Anastasi, 1988; Portney & Watkins, 1993). The individual SOS items having lowest reliability (paving glare and noise level) might have been influenced by time of day; similarly, the items related to indoor glare had shown low reliability in the TESS instrument. Adequate interrater and test–retest reliability findings for different facility types helped confirm that the SOS instrument can be reliably applied to a wide range of long-term care settings. Because this single tool can be used to assess outdoor spaces at multiple types of nondementia care settings, it will not substantially contribute to the growing “proliferation” of measures noted by Kane (2000, p. 520).

The items in the SOS tool were derived from multiple empirical sources and emphasize environmental support from the perspective of residents. This focus on behavioral “affordances” makes the instrument more adaptable to a diverse range of environmental settings and resident populations; the use of affordances may also be partly responsible for the reliability exhibited by the instrument. Table 2 shows the sources supporting the validity of the final SOS items, and whether they had high, medium, or low support, compared with other items. All items (100%) had appeared in the preliminary literature reviews, and all were supported by the subsequent survey of experts. A large percentage of the items (82%) were supported by resident preferences (categorized open-ended survey responses). Many items (53%) were also found to be significantly and positively associated with the targeted behavioral outcomes of outdoor usage and/or walking; none were negatively associated.

Limitations and Further Research

Instrument Design

This instrument is based on observable physical environment characteristics, rather than on the programs, policies, and staff attitudes that may influence outdoor usage in long-term care settings. The only policy issues directly reflected in the SOS tool are locked-door policies and maintenance, which can be observed through physical traces, and tend to have some temporal stability. Future research could develop a policy-related component to accompany this instrument. Because no comparable instrument evaluates outdoor features at this level of detail, concurrent criterion validity could not be examined; future instruments could provide comparable data for further assessments. As an instrument intended for a wide range of nonexpert stakeholders, the SOS tool does not assess physical environment characteristics that require specialized expertise to assess, such as configurational aspects (e.g., site layout, solar orientation, wayfinding, or path circulation), which require more site-specific data, and are difficult for nonexperts to analyze and interpret; future instrument development could explore whether additional experts-only items would make a positive addition to the tool.

Instrument Testing

The preliminary studies and final reliability testing were conducted within small samples in a single geographic region, due to funding limitations, whereas the multiregional study was conducted with a fairly large sample in a diverse range of climate regions. Although the preliminary and reliability studies addressed multilevel applicability by using a range of care levels, the multiregional study included only assisted living, due to constraints from the funding mechanism and shared study components. Future studies could use the SOS instrument to examine differences in reliability and item relevance in a wide range of facility types and geographic regions. Reliability was not estimated for several items added after the field-testing had begun; future studies could examine the reliability of these items. A 7-week test–retest interval was used; future studies could examine the stability of ratings over a longer time frame with greater seasonal changes, to determine whether the instrument is addressing relatively permanent environmental characteristics. Although interrater reliability testing was conducted with two raters, further testing could examine reliability with a large number of raters from different disciplines, including untrained raters and provider staff. If the results of different raters are to be compared, spaces should preferably be rated at the same season and time of day. If the chief goal is to evaluate the quality of outdoor space, and/or compare qualities at different facilities, ratings should be conducted in weather conditions comfortable for older adults. Future studies could also compare provider staff and researcher ratings at the same sites, to facilitate cross-disciplinary comparison. Further development and testing could determine whether a more streamlined self-scored format would be useful for nonresearchers and provider staff.

Future Scoring and Weighting

Pretesting and discussion with provider organizations revealed the importance of scoring to make the results more meaningful. The pilot version of the instrument did not include a scoring procedure for the overall instrument, making it difficult for users to understand the implications or to compare different outdoor spaces. Although aggregating scores for diverse items may lead to relatively high scores for facilities where a few excellent qualities overcompensate for other very poor aspects, the advantages of having summary scores are likely to outweigh the disadvantages. Developing a weighting system can help resolve this issue, by giving more weight to items that appear to be relatively essential, and have higher support for validity. Weighting would allow each item’s score to more proportionally represent its relative importance in supporting resident usage. In future research, a weighting metric (based on empirical data) should be developed, to facilitate comparison and provide scoring that more accurately reflects the supportive potential of specific environmental features.

Conclusion

The SOS instrument fills an important gap in assessing long-term care settings for older adults and can be used without special expertise to comprehensively evaluate outdoor environments as an important component of the residential environment. The tool can be shared among researchers, design professionals, provider organizations, and consumer advocates, allowing stakeholders to compare ratings and seek consensus based on the same instrument. Researchers can use the tool to compare health-related outcomes with environmental characteristics; design professionals can use it as a base for decision making; providers can use the tool to prioritize scarce budgetary resources for facility maintenance, remodeling, and new construction; consumer advocates can use the tool to compare specific health-promoting environmental characteristics among facilities. The SOS tool has strong implications for policy, practice, and research; it can be incorporated as a quality measure in traditional strategies such as licensing and certification; it can contribute to market-based strategies that reflect consumer demands for improved access to nature and opportunities for active aging; and it is likely to become an important measure in culture change strategies because of the strong emphasis on person-centered care and resident autonomy. For example, the SOS tool could be used to reinforce the Interpretive Guidelines used by the Centers for Medicare & Medicaid Services; it can help give facilities with high-quality outdoor access a competitive edge in the market. At the same time, the tool can promote an important goal of culture change, in emphasizing the need for independent access to outdoor areas, contrasted with many current facilities where residents must rely on staff to help them negotiate barriers to outdoor access. By increasing awareness of the environmental features associated with residents’ outdoor usage and preferences, the SOS instrument may enable future facilities to reduce barriers and increase the health-promoting potential of long-term care environments.

Acknowledgments

Yunhee Bae of the Texas A&M Measurement and Research Services assisted with data organization. Jennifer Black, Cristina Ferrara, Kaitlyn Legg, Adam Panter, and Katelyn Williams assisted in field-testing. Margaret Calkins, PhD, Lois Cutler, PhD, Rosalie Kane, PhD, Philip Sloane, MD, and James Varni, PhD provided valuable input on instrument development. We deeply appreciate the staff and residents of participating long-term care facilities, and the experts who provided input on the relative importance of environmental features.

Author notes

**

Decision Editor: Rachel Pruchno, PhD