Abstract

Background and objective Annotations to physical workspaces such as signs and notes are ubiquitous. When densely annotated, work areas become communication spaces. This study aims to characterize the types and purpose of such annotations.

Methods A qualitative observational study was undertaken in two wards and the radiology department of a 440-bed metropolitan teaching hospital. Images were purposefully sampled; 39 were analyzed after excluding inferior images.

Results Annotation functions included signaling identity, location, capability, status, availability, and operation. They encoded data, rules or procedural descriptions. Most aggregated into groups that either created a workflow by referencing each other, supported a common workflow without reference to each other, or were heterogeneous, referring to many workflows. Higher-level assemblies of such groupings were also observed.

Discussion Annotations make visible the gap between work done and the capability of a space to support work. Annotations are repairs of an environment, improving fitness for purpose, fixing inadequacy in design, or meeting emergent needs. Annotations thus record the missing information needed to undertake tasks, typically added post-implemented. Measuring annotation levels post-implementation could help assess the fit of technology to task. Physical and digital spaces could meet broader user needs by formally supporting user customization, ‘programming through annotation’. Augmented reality systems could also directly support annotation, addressing existing information gaps, and enhancing work with context sensitive annotation.

Conclusions Communication spaces offer a model of how work unfolds. Annotations make visible local adaptation that makes technology fit for purpose post-implementation and suggest an important role for annotatable information systems and digital augmentation of the physical environment.

Introduction

The communication space, where people interact with each other, exchanging messages and coordinating activities, is the largest part of any organization's information space.1 Interactions are necessarily shaped by the available communication channels,2 and every new wave of technology, from email to social media, can alter interaction dynamics.3

While the communication space is an abstraction, one part of it has a physical embodiment. We work and walk through different private, public, and professional spaces. Some of these are distinguishable because they are local hubs where people congregate and interact. One intriguing feature that distinguishes such hubs is that the physical environment is often modified over time by the addition of notes, messages, and signs. We might call such annotated environments communication spaces.

An annotation in such physical spaces could be any form of message—a permanent sign, image, text on a digital display, or a paper sticky note. These messages are often ignored, sometimes tattered, but are a near universal background to work. People who occupy a communication space are prompted, goaded, and guided by these messages to behave differently than they might elsewhere.

Communication spaces can be found in the common desk areas of hospital wards, around whiteboards in the operating theater and emergency department, or in the reception areas of primary care physicians. Documents and technologies can also create communication spaces. Clinicians may congregate over a patient chart or a computer screen. Communication spaces in organizations are often public commons and annotations social artifacts. Individuals can also annotate private spaces, for example with personal task lists.

Any study of such communication spaces rests on a profound legacy of scholarship. Human communication is theory rich.4 The study of signs and human engagement with the physical context is touched on by researchers in situated cognition,5 computer supported cooperative work and distributed cognition,6 and ubiquitous and context aware computing.7 Some research has explored how physical spaces shape information processing for clinical work. Whiteboards in emergency departments or in operating theaters, for example, are high activity places with unique characteristics that facilitate team coordination.8

Given that annotations represent a still largely unexplored class of information need, this paper reports a qualitative study of the ways in which annotations are used to support the work of clinical staff in a hospital setting. The analysis first leads to a typology of annotations by function, both individually and in the aggregate. It next generates a number of hypotheses about the underlying role of annotation in adapting space to task, and how information technology could be adapted to support annotation. Finally, as physical annotations could be substituted for by digital annotations in augmented reality (AR), hypotheses about the way this technology could be used in clinical settings are also presented.

Methods

Images of annotations were taken at a 440-bed metropolitan teaching hospital in Sydney, Australia. Images of annotated areas were purposefully sampled in two medical wards and the radiology department. This included ward corridors, common areas including areas for medication preparation, clerical areas, nurse stations, common staff rooms, hand-washing areas, and entrances to patient rooms. A set of 64 digital images was reduced to 39 after excluding technical inferior images where annotations could not be read or were not present. Approval for the study was granted by the University of New South Wales Ethics Committee.

A grounded theoretic approach was adopted to qualitatively develop a typology of messages, based on function.9 Categories were added until saturation was reached. The typology of messages was then used to generate hypotheses about the nature of annotation in communication spaces, as well as about the role of technology in supporting the information needs addressed by annotation.

Results

A typology of annotations

Annotations appeared to bind a space or its contents to different tasks or contexts. Six functional categories were identified, with annotations signaling identity, location, capability, status, availability, or operational methods (table 1). Spaces were also sometimes augmented with objects or tools such as hand wash bottles and surgical glove holders, bolted onto the walls of otherwise generic spaces (figure 1).

Table 1

Annotation types by functional role, and examples of each from the dataset

Annotation function Examples from data set 
Identity: Indicates the current binding of a location to an individual or task. Identity labels are ubiquitous, eg the name of a patient on a bedside binds bed to patient. ‘Bed 2’ label on door; hazardous substance sign on supply room door; sign on door indicating the patient inside is infectious (figure 1). 
Location: Landmarks and maps provide data needed to identify the immediate place or guide people to other places, eg signs with arrows and ward names. ‘Chart in pharmacy’ paper sheet to be inserted in a medical record (figure 4). 
Capability: Information labels or certificates make general statements about the quality, authority, or uses for a location, device, object or individuals, eg when a device was last tested. Certificates are use widely in other settings, eg academic qualifications on a wall to signify quality or accreditation. Sign ‘this extinguisher CO2—to be used for paint, oil, electrical and other liquid fires’; drawer label in cabinet ‘glucometer×2, torches on bottom shelf’. 
Status: Labels can indicate the current state of events within locations, eg ‘nil by mouth’ binds a patient to a clinical management state. An ‘out of order’ paper sign on a printer; color-coded stickers on x-ray paperwork awaiting report indicating urgency of request from urgent to routine. 
Availability: Signs can help individuals decide whether they can complete a task now, whether they should wait, or seek an alternate location or strategy, eg opening times for a pharmacy. Bed occupation status on whiteboard; sheet on wall ‘After hours support roster’ with times of availability and phone numbers of staff. 
Operation: Annotations can provide the information needed to select an action or execute a sequence of actions. A sign with notes on how to operate a clinical information system (figure 3) or how to wash hands (figure 1); wall chart in radiology explaining color coding of x-ray files eg, ultrasound, IVP. 
Annotation function Examples from data set 
Identity: Indicates the current binding of a location to an individual or task. Identity labels are ubiquitous, eg the name of a patient on a bedside binds bed to patient. ‘Bed 2’ label on door; hazardous substance sign on supply room door; sign on door indicating the patient inside is infectious (figure 1). 
Location: Landmarks and maps provide data needed to identify the immediate place or guide people to other places, eg signs with arrows and ward names. ‘Chart in pharmacy’ paper sheet to be inserted in a medical record (figure 4). 
Capability: Information labels or certificates make general statements about the quality, authority, or uses for a location, device, object or individuals, eg when a device was last tested. Certificates are use widely in other settings, eg academic qualifications on a wall to signify quality or accreditation. Sign ‘this extinguisher CO2—to be used for paint, oil, electrical and other liquid fires’; drawer label in cabinet ‘glucometer×2, torches on bottom shelf’. 
Status: Labels can indicate the current state of events within locations, eg ‘nil by mouth’ binds a patient to a clinical management state. An ‘out of order’ paper sign on a printer; color-coded stickers on x-ray paperwork awaiting report indicating urgency of request from urgent to routine. 
Availability: Signs can help individuals decide whether they can complete a task now, whether they should wait, or seek an alternate location or strategy, eg opening times for a pharmacy. Bed occupation status on whiteboard; sheet on wall ‘After hours support roster’ with times of availability and phone numbers of staff. 
Operation: Annotations can provide the information needed to select an action or execute a sequence of actions. A sign with notes on how to operate a clinical information system (figure 3) or how to wash hands (figure 1); wall chart in radiology explaining color coding of x-ray files eg, ultrasound, IVP. 
Figure 1

Annotations on a door form a tightly coupled set that supports infection control on entry and exit through the door.

Figure 1

Annotations on a door form a tightly coupled set that supports infection control on entry and exit through the door.

Like any information artifact, the annotations could have different informational roles, depending on whether they (1) contained data to support reasoning; (2) codified declarative knowledge or rules; or (3) captured procedural knowledge, explaining how tasks are to be executed (table 2).

Table 2

Examples of annotations by informational role and scope of application

 Local scope General scope 
Data: Tags of identity, availability, capability, status, and location all provide context specific data that can support reasoning in a space. Drug labels on trays in a medicines trolley; a paper sheet in a clerking space lists the phone numbers of selected staff and frequently required locations and services, eg bookings, private CT, tea room; paper sheets pinned to a desk in radiology list patient names, procedures, times, and investigation status; a hand annotated chart on a board in central ward area wall with patient beds, associated times for nursing medication administration or observations; whiteboard with admitted patients by room, attending clinician, and status eg, ‘CABG ? WED’. Printed information sheets found on walls and noticeboards: ‘Consumer and carer rights an responsibilities' information sheet’; ‘why do we keep asking you who you are?’; ‘Medical nursing imaging—career information poster’. 
Rules: Declarative statements saying how things must be, but not how to comply with the demand. Some rules are obligatory, ‘All visitors must wash hands’, and others conditional, ‘If you have a cough or fever please notify reception’. Outside patient room—‘STOP—if you have cold or flu symptoms do not enter; In clinical clerking area—‘please put the notes back in the right place’; on top of computer terminal—‘emR—Protect your security—change user or exit’. On ward corridor wall—‘Attention—if you or your family have been in contact with someone who has had chicken pox in the last 3 weeks please notify staff immediately’; ‘wash your hands’. 
Procedures: Procedural messages contain a method needed to complete a task. Patient contact procedure outside infectious patient's room indicates how to enter room and interact with patient; computer login procedures and sheets explain the ‘electronic medical record, what you need to know’, eg allergies recording, attached to wall next to a computer terminal; ‘6 simple steps: respiratory hygiene for staff of healthcare facilities' outside isolation room; sheet with the times, duration, and rules for nursing handover, eg ‘PM staff will handover to the PM I/C by 2120 h—NO LATER’. Procedures for bathing and lifting patients in different categories of mobility on ward corridor wall; leaflets on staff noticeboard—newsletter with information on patient safety, eg ‘time out procedure’; memorandum from clinical chemistry department on change in procedures for measurement of serum troponin; sheets on falls prevention and falls management procedures. 
 Local scope General scope 
Data: Tags of identity, availability, capability, status, and location all provide context specific data that can support reasoning in a space. Drug labels on trays in a medicines trolley; a paper sheet in a clerking space lists the phone numbers of selected staff and frequently required locations and services, eg bookings, private CT, tea room; paper sheets pinned to a desk in radiology list patient names, procedures, times, and investigation status; a hand annotated chart on a board in central ward area wall with patient beds, associated times for nursing medication administration or observations; whiteboard with admitted patients by room, attending clinician, and status eg, ‘CABG ? WED’. Printed information sheets found on walls and noticeboards: ‘Consumer and carer rights an responsibilities' information sheet’; ‘why do we keep asking you who you are?’; ‘Medical nursing imaging—career information poster’. 
Rules: Declarative statements saying how things must be, but not how to comply with the demand. Some rules are obligatory, ‘All visitors must wash hands’, and others conditional, ‘If you have a cough or fever please notify reception’. Outside patient room—‘STOP—if you have cold or flu symptoms do not enter; In clinical clerking area—‘please put the notes back in the right place’; on top of computer terminal—‘emR—Protect your security—change user or exit’. On ward corridor wall—‘Attention—if you or your family have been in contact with someone who has had chicken pox in the last 3 weeks please notify staff immediately’; ‘wash your hands’. 
Procedures: Procedural messages contain a method needed to complete a task. Patient contact procedure outside infectious patient's room indicates how to enter room and interact with patient; computer login procedures and sheets explain the ‘electronic medical record, what you need to know’, eg allergies recording, attached to wall next to a computer terminal; ‘6 simple steps: respiratory hygiene for staff of healthcare facilities' outside isolation room; sheet with the times, duration, and rules for nursing handover, eg ‘PM staff will handover to the PM I/C by 2120 h—NO LATER’. Procedures for bathing and lifting patients in different categories of mobility on ward corridor wall; leaflets on staff noticeboard—newsletter with information on patient safety, eg ‘time out procedure’; memorandum from clinical chemistry department on change in procedures for measurement of serum troponin; sheets on falls prevention and falls management procedures. 

Relationship of annotations to their physical context

Annotations varied in their scope of application (table 2). Some had a highly local scope, for example a patient name on a bed. Others applied widely, potentially across the entire organization, for example an information sheet for patients outlining legal rights and responsibilities. Transit areas like corridors appeared to mainly carry messages with wide scope. Some annotations additionally had a situational scope, for example emergency exit signage becomes valuable in emergency situations.

Communication spaces are annotation assemblies

Annotations in the dataset typically collected together in assemblies, and finding an individual annotation alone in a physical space was the exception. These assemblies could be distinguished both by the degree to which the annotations in the group were coupled to common workflow, and whether the workflow is within or external to the space (figure 2):

Figure 2

Annotation assemblies are distinguishable by the degree to which they refer to the elements of the space within which they are found, and the degree to which they refer to external workflows. Some communication spaces are composites, assembled out of a number of discrete subassemblies.

Figure 2

Annotation assemblies are distinguishable by the degree to which they refer to the elements of the space within which they are found, and the degree to which they refer to external workflows. Some communication spaces are composites, assembled out of a number of discrete subassemblies.

Tightly internally coupled: Some annotation sets were clearly interdependent. For example, a door leading into a restricted-entry patient room was annotated with the room number, a poster with specific infection control procedures triggered on room entry and exit, including hand washing, and a bottle of hand wash, itself ‘annotated’ to the door (figure 1). Together these annotations constructed a workflow that said ‘at this location, use this procedure, with this tool‘. Generic wall spaces and columns were also turned into infectious precaution spaces that contained holders for surgical gloves and gowns, a hand wash bottle, and the instructions in their use.

Tightly externally coupled: Some annotation sets were indirectly coupled, each referencing different parts of a common external workflow. For example, the annotations at computer workstations referenced different functions in the clinical software (figure 3). A medication trolley was annotated by nursing staff to reflect their local workflow (figure 4). Medications held in stock on the ward were sorted into hand-labeled trays by medication class (eg, ‘analgesics’, ‘antihypertensives’). Medications ordered from pharmacy for specific patients were put in trays labeled by bed number. These annotations were design elements added by staff to shape the execution of the local workflow.

Figure 3

Annotations in the space surrounding hospital computer workstations contain information to assist in operating the clinical software. Clockwise from left: (1) a list of software icons and their associated functions on the wall next to a computer screen; (2) a note on top of a screen reminding users to ‘Protect your security’, followed by the icons to ‘Change user’ or ‘Exit’; (3) paper sheets on a wall behind a workstation with detailed instructions: ‘Electronic Medical Record–what you need to know’, eg ‘Allergies Recording’ followed by detailed instructions.

Figure 3

Annotations in the space surrounding hospital computer workstations contain information to assist in operating the clinical software. Clockwise from left: (1) a list of software icons and their associated functions on the wall next to a computer screen; (2) a note on top of a screen reminding users to ‘Protect your security’, followed by the icons to ‘Change user’ or ‘Exit’; (3) paper sheets on a wall behind a workstation with detailed instructions: ‘Electronic Medical Record–what you need to know’, eg ‘Allergies Recording’ followed by detailed instructions.

Figure 4

Annotations on a medication trolley: (1) the trolley carries sheets to be used in special circumstances, eg : the ‘Chart in Pharmacy’ paper sheet which acts as a placeholder for a patient chart whilst it is off ward; drug trays are annotated with handwritten labels: (2) patient bed numbers for patients receiving medications that are special orders from Pharmacy (eg ‘Bed 5+6’); and (3) common medications stored on the ward are organized by general class (eg ‘Antibiotics’, ‘Analgesia’, ‘Cardiac Meds’).

Figure 4

Annotations on a medication trolley: (1) the trolley carries sheets to be used in special circumstances, eg : the ‘Chart in Pharmacy’ paper sheet which acts as a placeholder for a patient chart whilst it is off ward; drug trays are annotated with handwritten labels: (2) patient bed numbers for patients receiving medications that are special orders from Pharmacy (eg ‘Bed 5+6’); and (3) common medications stored on the ward are organized by general class (eg ‘Antibiotics’, ‘Analgesia’, ‘Cardiac Meds’).

Loosely externally coupled: Spaces such as noticeboards contained a heterogeneous set of annotations that did not cohere internally, and referenced multiple separate workflows external to the space, but common to the ward (figure 5).

Figure 5

A noticeboard in a common ward clerking area aggregates a heterogeneous set of annotations all referencing different processes external to the space. Clockwise from top left: (1) a hand annotated paper sheet with a table of patient bed by hour of the day, indicating the observations taken by a nurse; (2) one of three sheets describing ‘Falls prevention strategies' for the hospital; (3) typed sheet with times and procedures for clinical nursing handover in the ward; (4) a district wide newsletter describing patient safety procedures, eg ‘Time Out’.

Figure 5

A noticeboard in a common ward clerking area aggregates a heterogeneous set of annotations all referencing different processes external to the space. Clockwise from top left: (1) a hand annotated paper sheet with a table of patient bed by hour of the day, indicating the observations taken by a nurse; (2) one of three sheets describing ‘Falls prevention strategies' for the hospital; (3) typed sheet with times and procedures for clinical nursing handover in the ward; (4) a district wide newsletter describing patient safety procedures, eg ‘Time Out’.

Multispace assembly: Some communication spaces had second order structure, in essence being an assembly of assemblies. A clerking area in a ward (figure 6) was assembled from a number of clearly distinct elements including an annotated computer workstation and a noticeboard, as well as general filing and workbench elements.

Figure 6

This communication space in a hospital ward clerking area is a composite of several distinct spaces: (1) a noticeboard (figure 5); (2) a filing area; (3) a computer workstation (figure 3, left-hand side); and (4) a workbench.

Figure 6

This communication space in a hospital ward clerking area is a composite of several distinct spaces: (1) a noticeboard (figure 5); (2) a filing area; (3) a computer workstation (figure 3, left-hand side); and (4) a workbench.

Discussion

Annotations as missing information

It is likely that there is a rich typology of annotations beyond that described here, and that their types, frequency, and distribution will vary with different environments and the individuals that occupy them. However as a group, the existence of annotation in work spaces indicates that they are put there to meet a distinct set of information needs, owned mainly by the users of an environment rather than those that designed it. Annotations in this dataset appeared to exist for four main reasons:

  1. The space provided insufficient information on how to complete a task or use a tool for the current users of a space (eg, a printer is not obviously malfunctioning on inspection, so a paper sign is needed; the computer workstation in figure 3 is annotated with the meaning of software icons because their function is not obvious).

  2. There is a need to support local variations in workflow by customizing a space to better fit local practice (eg, annotations on the medication trolley in figure 4).

  3. There is a need to restrict local variation in workflow by ‘patching’ spaces or tools to prevent unexpected user behaviors, or to encourage behaviors that are not happening despite the design's intent (eg, the hand washing cluster in figure 1, or the reminder to sign out the clinical information system on top of the computer terminal in figure 3).

  4. Spaces or tools are repurposed for uses not initially specified by their designers (eg, the door annotations in figure 1 are needed because a general purpose patient room was redesignated as an isolation room).

Annotations are thus not just workarounds or short cuts. Every annotation appears to make a space more fit for someone's purpose. We might think of annotations as repairs to an environment, improving its fitness for current purpose, fixing inadequacy in design, or meeting emergent or unanticipated needs. Annotations thus appear to contain missing information needed for tasks to be executed in a specific work environment by its current occupiers. Implementation has long been seen as a process of mutual adaptation of technology and organization.10 The act of annotation or repair is therefore one of unfolding or emergent design11 that occurs post-implementation through user customization.12

Annotations help make visible the gap between work that is done in a space and the capability of a space to support that work. The notes, manuals, and instructions that annotate a computer workstation make explicit the gap between the information system workflow and the needs and capabilities of those that use it. There is thus a direct analogy between the concept of task–space mismatch, as measured by annotations and missing information, and the notion of task–technology fit.13,14 The task–space gap also represents a gap between the mental models of a space's users and those of its designers, a gap in their shared common ground.1 Annotations can thus be thought of as grounding or communication acts, more often made by users than designers, to close this gap in common ground.

Measuring missing information

The notion that annotations capture missing information needed to complete a task, and that this information is a measure of the gap between a technology and the task it was designed to support, is a testable hypothesis.

Given a work environment, a set of users and tasks, we would expect to see a difference in missing information when any two of these are held steady and a third is varied. For example, if we compared annotations in two workplaces of similar design and purpose, created by staff with different experience or training, then we might expect annotations to feature more prominently with the less experienced group because of their greater informational needs. Equally, if there were a drift apart in the tasks of otherwise similar environments and workers, we would predict annotations would track the shift in tasks. Given that the success of a technology or practice is often highly dependent on local context, annotations may be a powerful clue to the source of that variation. Assessing the nature, number, and trajectory of user repairs should provide us with an estimate of the dynamic fitness of any tool or technology at a give place and moment.

Information theory could provide a simple approach to quantifying annotation,15 measuring the length of the messages by number of characters. This however provides no insight into the purpose of messages, or how they are used. For example, the length of a message varies with the common ground shared between message creator and receiver.3 One would expect shorter messages when there is high common ground and longer ones when there is not. Such a metric is also independent of the frequency of use of an annotation. To measure the effect of annotation we need to know the frequency of the task for which it is designed, and the frequency of engagement with the annotation when completing that task. If annotations are not used then they are not meeting a need.

An alternate approach to quantification measures message content via the cognitive effort involved in enacting it. A classic approach to measuring cognitive task complexity is to break tasks down into unit actions or elementary information processes (EIPs) (eg, read a value, compare two values, delete a value).16,17 Using this framework, two annotations that contain a single datum would be considered of identical complexity as both require a single read operation (1 EIP) (and both correspond to the environment missing a single datum). In contrast, rules or procedures necessarily require a sequence of acts and are of higher complexity. For example, a sign showing the time a service is available requires the sign to be read, the reader to then check the current time to get a second datum, and to then compare the two values before acting (3 EIPs).

The EIP concept underpins the classic GOMS (goals, operators, methods, and selection rules) framework,18 and the tools that use GOMS to estimate software useability.19 By logging user actions as they engage with software, methods such as GOMS permit quantification of effort required to use software. Although it may be methodologically less easy to track user actions in physical space compared to software, this approach is likely to be close to that needed when measuring engagement with annotation.

Creating communication spaces by design

The most direct response to observing annotation is to support the information need implied by it. If we find telephone numbers on walls, there may be a requirement for improved directory services. If we find notes about how to use a piece of equipment, it is a signal that a device needs clearer design, repair, or perhaps that staff training is inadequate.

A more general approach is to recognize that annotations are a natural response of individuals to their environment, and are often emergent responses as tasks or users change. We could thus explicitly add a capability for annotation to our spaces, allowing them to better adapt to changing circumstance. For example, the patient bedside is already a communication space,20 but may not be seen as such. Intentionally designing the bedside as a communication space would see existing annotations like signs and notes be replaced with tools designed to support a broader set of annotations. Patients can have their questions or concerns visible for all to see rather than being left unsaid, making the space around a bed ‘their’ space. Information technology can be considered as an additional layer of ‘annotation’ over a physical communication space. Active signs by the bedside can display patient name, their treating clinician, messages such as dietary restrictions, that the patient is currently elsewhere having a test done, or resuscitation orders. Portable digital devices can display location specific information when they are in the vicinity of a bed, for example any alerts or warnings about a patient.

More generally, the workplace can now be conceived of as a set of interleaved communication spaces. Different task-specific spaces can be aggregated into larger ones, creating support for a variety of tasks and user groups (figure 6). Some communication spaces, like corridors, are transitional or boundary spaces, and have no strong local task context. They naturally lend themselves to tools that support a wide variety of message types. Not only would these tools support the messages available today, whose scope is general or organization wide, messages could be crafted to those currently transiting the space. Just as advertisers hope to use individual identification technologies to create personalized messages, organizational areas can narrowcast tailored messages when they know who is within them. Such technologies should allow communication spaces to be engineered in highly personalizable ways.

What communication spaces may tell us about digital system design

Instead of forcing users to stick physical post-it notes onto digital displays, the capability to annotate and customize can be embedded in software21,24 allowing the capture of tags, notes, and reminders. Annotation is increasingly proposed as a mechanism for engagement within information systems, for example allowing users to annotate shared maps,23 information objects in social media spaces,25,26 and they may also have clinical utility.27

Clinical information systems today are not intentionally designed as annotatable spaces. Point to point communication, such as alerts and reminders, is possible but there is no easy way to directly annotate workflow elements (such as menu or file structures) or the data itself. An electronic health record (EHR) may allow free text notes at specific locations, yet with ‘old fashioned’ paper patient records, no one is stopped from attaching a note to the top sheet, or a specific location on any page. We must put messages in an EHR only where we are allowed, not where we want.

What is lacking in most proposals for digital annotation systems is any analysis of the way annotation unfolds in non-digital spaces, and as a result the systems offered allow only for the attachment of generic text notes. Yet in the preceding analysis, it became clear that annotations have different purposes, signaling anything from identity to operational status. If so, then there should be value in defining a range of annotation types, which can link directly to different digital objects within an information system. If a note that a patient is ‘nil by mouth’ on a digital display was semantically typed, it could trigger actions, such as canceling a meal order.

Another interesting feature of the current dataset is the way that multiple annotations appeared to work as a group to support a workflow. Analysis of how communal whiteboards are used to coordinate or schedule clinical tasks has also revealed a rich language and higher level logic to the annotations required to do such coordination.8 This suggests that compositionality is another important property we could incorporate into digital annotation. Annotations could refer to other annotations, and together compose meaningful higher order ‘sentences' (eg, ‘at this location, use this procedure, with this tool’). If such compositions are semantically typed they can have computational consequence, and not just passively signal information to a reader.28

Current approaches to customization of clinical information systems tend to emphasize configuration by template, often by formal request to central organizational experts. When clinicians are offered the opportunity to configure information displays, they appear to become far more effective in retrieving and reasoning with information.29 Programming by annotation appears to be a further natural extension to user customization, giving users a simple local programming language to tailor or create work processes through annotation by crafting meaningful, computationally active, work fragments.

It should be possible to test the benefits of this model of user programming. We would hypothesize that any well designed annotation system first would see frequent use, as workers replace annotations in physical space with ones in digital environments. Second, we would expect digital communication spaces to be more resilient to change in work practice or work force, given that their intent is to allow users to customize systems to their local needs.

From communication spaces to augmented realities

Digital annotation need not occur only in ‘virtual’ information spaces. AR technology30,31 allows digital annotation to overlay physical space. AR can be achieved using personal head-up displays, digital spectacles, or handheld devices like camera-enabled smart phones. It is also possible to make physical space directly augmentable by building it out of digital surfaces that can respond to input modalities like touch and gesture.32,33

AR appears an ideal vehicle to support our need to annotate space. The applications for AR annotation in clinical practice may be extensive. It is well known for example, that patients are at risk of iatrogenic harm, and reducing the rates of errors remains an elusive goal.34 Prompts and reminders in augmented space could be used to do anything from encouraging hand washing to preventing wrong-side surgery (box 1). Augmentations need not be static, but can also encode procedures.35 For example, using image recognition software to identify anatomical landmarks, a surgical site can be annotated with instructions for each stage of a surgery,36 highlighting where work must be done, what instruments are needed, and relevant physiological data. A patient's arm can similarly be annotated with anatomical markings to assist with insertion of an intravenous line, note areas of injury to be handled carefully, or mark the time an injection was previously given or is next due.

Box 1

The augmented hospital—examples of digital patient safety annotations over physical space

Data:

  • Identifying labels on staff and patients.

  • Correct side label for surgical procedures.

  • New tests results available, eg on patient, patient bed, or on entering a record system.

  • Time of last or next scheduled medication, or procedure, eg change position in bed.

  • Allergy or dosage warnings on patients or medications.

  • Medical Emergency Team activation criteria status.

Rules:

  • Resuscitation rules for a patient, eg Do not resuscitate.

  • Failure to hand wash alerts that appear when entering a patient space, or at a hand washing station.

Procedures:

  • Pop-up instructions on devices, eg how to set up a specific dosing regime on an intravenous pump, or a visualization of the order that has just been entered into the pump to double check the order is correct.

  • Surgical procedure prompts highlighting anatomical landmarks, next steps, tools required, patient state, eg oxygen saturation.

Given the capacity for humans to work in complex communication spaces which are assemblies of smaller spaces, the design of AR enabled spaces could create different spaces in the visual field to accommodate different tasks sets, for example a personal task annotation area, a communication area, and a current task focus area. Further individuals would only see the annotations they needed to. What today is on view to everyone, can be replaced in AR by highly targeted labels.

For this hypothesized role for AR in clinical practice to be established there would need to be evaluations both with individual clinicians, to ensure visual annotations do not interfere with work or lead to harm or error, as well as at the level of the organization, to demonstrate that the anticipated benefits to patients can be realized. This approach can also be used in simulated virtual worlds for training purposes.37 Clinicians can move through virtual spaces, interacting with virtual patients, being reminded by annotations when they fail to wash their hands, or carry out actions that carry risk.38

It remains an open question about the degree to which physical space can be digitally augmented and remain workable, and what the role and language of annotation should be. The annotation typology presented here offers a first-cut of that language. We are fortunate that we have an annotated world from which to draw inspiration and which provides a working model, as well as a growing literature on interaction design in digital space to help transform physical annotation into dynamic digital augmentation.

Limitations

This qualitative study was designed to support hypothesis generation about the nature and uses of annotations, and not for hypothesis testing. Data sampling was purposive, seeking to build a rich picture of the variety of annotations found, but the sample is neither representative nor exhaustive. The dataset may have missed examples that suggest additional categories or roles for annotations at the study site, and other sites are likely to extend our understanding further. It is not possible with this data to make comments about the frequencies or distributions of annotations, nor to quantify content in any way. The data does not include any formal documentation of workflows, so it is also not possible to directly correlate annotations and work.

Future work

Future work should focus on developing a more representative typology of annotations in different settings, as well as conducting more comprehensive analyses at individual sites. The challenge of measuring annotation will require efforts to validate either the application of existence measures of cognitive processing, or the development of new measures more attuned to the nature of annotation.

Communication spaces can be seen as dynamic and evolving ecologies, in the same way that any man–machine system is ecologic.39,40 Studying how we interact with annotation should reveal how frequently we engage with or ignore annotation, how frequently we need to use annotation to accomplish specific tasks, and how performance alters because of annotation. Analysis of the rate of annotation turnover, and whether messages are well curated or abandoned may tell us about how a locale's users and their needs and tasks change. Changes in annotation location, content, and density over time could tell us much about the work that happens in a space. A set of annotations should be evidence that a space has a unique identity or role, and may map its boundaries and the way it is used.

Conclusions

Communication spaces provide us with a conceptual model to describe the way clinical work is organized. Annotations present an ethnographic opportunity to make ‘work visible’,41 and appear to teach us much about system design and implementation, given that they appear after system implementation.42 A physical work space is a situated cognition ‘engine’, where the knowledge contained in annotations tells us something about how tasks are to be executed in a specific place—the ‘logic of here’.43

The lessons we learn from the way humans annotate physical spaces to meet their needs have clear digital analogs, and the technical proposals outlined here seem a promising way to take clinical information system design in a different direction. It represents an opportunity to move away from our current top-down and document centric world of forms and records, towards one that is more user-driven and focused on physical rather than informational workflows. The future of human work is not like the present, where information technology and place are ignorant of each other. When technology and place mutually shape each other, and when we have some direct and natural say in that shaping through annotation, we may move to a very different mode of working.

Acknowledgements

Photographs were taken by Mei-Sing Ong.

Contributors

EC conceived of and wrote the paper, and is its guarantor.

Funding

The author acknowledges the support of Australian Research Council Linkage Grant LP0775532 and the National Health and Medical Research Council (NHMRC) Centre for Research Excellence in E-Health. The study sponsors had no role in study design; in the collection, analysis and interpretation of the data; in the writing of the manuscript; or in the decision to submit for publication.

Competing interests

None.

Ethics approval

University of New South Wales.

Provenance and peer review

Not commissioned; externally peer reviewed.

Data sharing statement

Digital images used in this study are covered by ethics committee requirements that protect the privacy of the organization they were taken from. Some images contain patient or staff identifiers and so public access is currently not available.

References

1
Coiera
E
.
When conversation is better than computation
.
J Am Med Inform Assoc
 
2000
;
7
:
277
86
.
2
Coiera
E
.
Interaction design theory
.
Int J Med Inform
 
2003
;
69
:
205
22
.
3
Coiera
E
.
Mediated agent interaction
. In:
Quaglini
BA
, ed.
8th conference on artificial intelligence in medicine
 .
Berlin
:
Springer Lecture Notes in Artificial Intelligence No 2101
,
2001
:
1
15
.
4
Littlejohn
SW
Foss
KA
.
Theories of human communication
 .
9th edn
.
Wadsworth
,
2007
.
5
Suchman
L
.
Plans and situated actions: the problem of human machine interaction
 .
Boston, MA
:
Cambridge University Press
,
1987
.
6
Xiao
Y
.
Artifacts and collaborative work in healthcare: methodological, theoretical, and technological implications of the tangible
.
J Biomed Inform
 
2005
;
38
:
26
33
.
7
Bricon-Souf
N
Newman
CR
.
Context awareness in health care: a review
.
Int J Med Inform
 
2007
;
76
:
2
12
.
8
Prinz
W
Jarke
M
Rogers
Y
et al
. eds.
Cognitive properties of a whiteboard: a case study in a trauma centre
.
Proceedings of the Seventh European Conference on Computer-Supported Cooperative Work
.
Bonn, Germany
:
Kluwer Academic Publishers
,
2001
.
9
Corbin
J
Strauss
A
.
Basics of qualitative research: techniques and procedures for developing grounded theory
 .
Sage Publications
,
2007
.
10
Leonard
DA
.
Implementation as mutual adaptation of technology and organization
.
Research Policy
 
1988
;
17
:
251
67
.
11
Rice
RE
Rogers
EM
.
Reinvention in the innovation process
.
Sci Commun
 
1980
;
1
:
499
514
.
12
Bogers
M
Afuah
A
Bastian
B
.
Users as innovators: a review, critique, and future research directions
.
J Manag
 
2010
;
36
:
857
75
.
13
Goodhue
DL
Thompson
RL
.
Task-technology fit and individual performance
.
MIS Quarterly
 
1995
:
213
36
.
14
Dishaw
MT
Strong
DM
.
Extending the technology acceptance model with task–technology fit constructs
.
Info Manage
 
1999
;
36
:
9
21
.
15
Shannon
CE
Weaver
W
.
A mathematical theory of communication: American telephone and telegraph company
 ,
1948
.
16
Chase
WG
.
Elementary information processes
. In:
Estes
WK
, eds.
Handbook of learning and cognitive processes
 . Hillsdale, NJ: L Erlbaum Associates; New York
1978
:
19
90
.
17
Sintchenko
VS
Coiera
E
.
Which clinical decisions benefit from automation? A task complexity approach
.
Int J Med Inform
 
2003
;
70
:
309
16
.
18
Card
SK
Moran
TP
Newell
A
.
The psychology of human computer interaction
 .
Routledge
,
1983
.
19
John
BE
Kieras
DE
.
The GOMS family of user interface analysis techniques: comparison and contrast
.
ACM Transactions on Computer-Human Interaction
 
1996
;
3
:
320
51
.
20
Caligtan
CA
Carroll
DL
Hurley
AC
et al
.
Bedside information technology to support patient-centered care
.
Int J Med Inform
 
2012
;
81
:
442
51
.
21
DiGiano C, Tatar D, Kireyev K. Learning from the Post-It: Building Collective Intelligence through Lightweight, Flexible Technology. Conf. suppl. ACM SIG Conf. Computer-Supported Cooperative Work (CSCW 2006), ACM Press, 2006:65–66.
22
Looi
CK
Chen
W
Ng
FK
.
Collaborative activities enabled by GroupScribbles (GS): an exploratory study of learning effectiveness
.
Comput Educ
 
2010
;
54
:
14
26
.
23
Fagerberg
P
Espinoza
F
Persson
P
.
What is a place? allowing users to name and define places
.
CHI’03 extended abstracts on human factors in computing systems
.
2003
:
828
29
.
24
Patterson HR, Ribarsky WW, Bolter J. The virtual annotation system. Virtual Reality Annual International Symposium, IEEE, 1996:239–45.
25
Li
R
Bao
S
Yu
Y
et al
.
Towards effective browsing of large scale social annotations
.
Proceedings of the 16th international conference on World Wide Web
.
2007
:
943
52
.
26
Helic
D
Trattner
C
Strohmaier
M
et al
.
On the navigability of social tagging systems
.
Social Computing (SocialCom), 2010 IEEE Second International Conference
.
2010
:
161
68
.
27
Bringay
S
Barry
C
Charlet
J
.
Annotations for the collaboration of the health professionals
.
AMIA Annual Symposium Proceedings
.
2006
:
91
.
28
Handschuh S, Staab S, Volz R. On deep annotation. Proceedings of the 12th international conference on World Wide Web. ACM, 2003:431–8.
29
Senathirajah
Y
Kaufman
D
Bakken
S
.
The clinician in the driver seat: cognition and interaction in MedWISE
.
AMIA Annu Symp
 
2012
:
1653
.
30
Azuma
RT
.
A survey of augmented reality
.
Presence: Teleoperators and Virtual Environments
 
1997
;
6
:
355
85
.
31
Azuma
R
Baillot
Y
Behringer
R
et al
.
Recent advances in augmented reality
.
IEEE Comput Graph Appl
 
2001
;
21
:
34
47
.
32
Wobbrock JO, Morris MR, Wilson AD. User-defined gestures for surface computing. Proceedings of the 27th international conference on Human factors in computing systems, ACM, 2009:1083–92.
33
Piper AM, Campbell R, Hollan JD. Exploring the accessibility and appeal of surface computing for older adult health care support. Proceedings of the 28th international conference on Human factors in computing systems, ACM:907–16.
34
Braithwaite
J
Coiera
E
.
Beyond patient safety Flatland
.
J R Soc Med
 
2010
;
103
:
219
25
.
35
Feiner
S
Macintyre
B
Seligmann
D
.
Knowledge-based augmented reality
.
Commun ACM
 
1993
;
36
:
53
62
.
36
Shuhaiber
JH
.
Augmented reality in surgery
.
Arch Surg
 
2004
;
139
:
170
4
.
37
Doyle
P
Hayes-Roth
B
.
Agents in annotated worlds
.
Proceedings of the second international conference on Autonomous agents
.
1998
:
173
80
.
38
Jensen
AMD
Jensen
MM
Korsager
AS
et al
.
Using virtual worlds to train healthcare workers—a case study using Second Life to improve the safety of inpatient transfers electronic
.
J Health Informatics
 
2011
;
7
:
e7
.
39
Vicente
KJ
Rasmussen
J
.
Ecological interface design: theoretical foundations
.
IEEE Transactions on Systems, Man and Cybernetics
 
1992
;
22
:
589
606
.
40
Gaver
WW
.
Situating action II: affordances for interaction: the social is material for design
.
Ecological Psychology
 
1996
;
8
:
111
29
.
41
Suchman
L
.
Making work visible
.
Commun ACM
 
1995
;
38
:
56
64
.
42
Coiera
E
.
Getting technical about socio-technical systems science
.
Int J Med Inform
 
2007
;
76
:
S98
S103
.
43
Suchman
L
.
Human-machine reconfigurations: plans and situated actions.
 
Boston, MA
:
Cambridge University Press
,
2006
.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (http://creativecommons.org/licenses/by-nc-nd/3.0/) which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contact journals.permissions@oup.com

Comments

0 Comments