Who Controls Who? Embodied Control Within Human–Technology Choreographies

Abstract

In this paper, we explore issues of embodied control that relate to current and future technologies in which body movements function as an instrument of control. Instead of just seeing ourselves in control, it is time to consider how these technologies actually control our moving bodies and transform our lived spaces. By shifting the focus from devices to choreographies among devices, we perform a theoretical analysis of the multidimensional aspects that reside within embodied interaction with technology. We suggest that it is beneficial to acknowledge and reformulate the phenomena of embodied control that go beyond the instrumental user-to-device control scheme. Drawing upon the phenomenology of the body, ecological psychology and embodied cognitive science, we identify three different dimensions of embodied control: instrumental, experiential and infrastructural. Design implications of this theoretical model are also discussed.

RESEARCH HIGHLIGHTS

• This paper clarifies the roles of technology and human beings in terms of movements and control: how we use movements in the control of technology, and reciprocally, how our movements are controlled by technology, thus constituting technology-induced choreographies.

• The theoretical framework presented promotes the viewpoint of phenomenological philosophy on human–technology studies. This is done especially by treating movements as lived experiences that constitute humans as embodied beings.

• The analysis of embodied control allows researchers and practitioners to be mindful in their visions and conceptualizations of body movement and control among different human–technology assemblages.

• For human–computer interaction-related empirical research and practical design, this study formulates new ontological and epistemological groundings as well as new conceptual tools.

• For interaction design, this paper underlines and illustrates the power and responsibilities embedded in developing instruments of control that contribute to choreographing human movements.

1. INTRODUCTION

The common rhetoric of technological discourse has emphasized recurrently how new gadgets and gizmos will transform everyday life in the near future. Although our personal and public surroundings are already filling up with various fancy gadgets and gizmos, we rarely start to reflect on how our engagement with all those technologies actually changes our everyday embodied living. In this paper, we look at some of this transformational potential of technology by discussing how implementations of so-called smart technology affect our bodily flow of everyday activities and movements—that is, routines and everyday choreographies we regularly engage in, but of whose real contents and embedded meanings we rarely trouble ourselves to become aware.

Our surroundings consist of built architectures, pieces of furniture, devices, as well as rituals of behavior and other aspects that have an effect on how we move around. Instead of seeing these surroundings merely as a ‘neutral, external backdrop to human activities’ (Ingold, 1993, p. 152), we consider surroundings as temporal continuums, regarding them as spaces of possible practices and engagement that are constituted through the activities of dwelling in them (Ingold, 1993). Within such temporal continuums, our everyday actions are constantly choreographed through engagements with both the natural and the technologized surroundings. In line with ecological psychology, one can consider these effects on movements in terms of action affordances, referring to perceived possibilities of action offered or denied by the environment (Gibson, 1979). Within a conceptual framework of choreographies, we can also understand them as pre-choreographies (Parviainen et al., 2013a) or as choreographic inscriptions (Loke and Kocaballi, 2016), both referring to elements in design (material and social) that constitute movement configurations, either purposely or as an unintended side effect. Technological artifacts and user interfaces do not exist in a vacuum. Rather, they are inevitably fused into the context of a more general physical and social infrastructure that facilitates, triggers, guides and orientates the dynamics of everyday movements. Therefore, emergence of choreographies within the human–technology relationship is not simply a matter of how routines become accommodated with each new piece of technology, because each piece of technology potentially has an effect on the whole infrastructure that defines everyday encounters with spaces and environments (see Dourish and Bell, 2007).

Despite the ever-growing concern regarding contextual factors and everyday life related to interaction design (e.g. Arthur, 2010; Beyer and Holtzblatt, 1998; Carroll, 2000; Savat, 2013), the predominant stance is still more or less focused on the use of devices or applications. The use-oriented undercurrent in thinking is perfectly understandable, considering the roots of human–computer interaction (HCI) and the fact that the designers’ business is ultimately in developing discrete products for the markets. According to Harper (2003, p. 22), the adaption of digital technologies into homes can generate various conflicts with ‘appropriate behavior’ for different times and places. The home is an example of a place for the plurality of occasions, ranging from resting to hosting friends or living out personal desires to sharing space with others who may have different desires. Therefore, the home is not a singularity for which straightforward needs for using technology could be defined. It may be easier to delineate such needs for more particular contexts of professional tasks, but the ways that technological devices are marketed to consumers are largely a technology-push rather than demand-pull (e.g. Blanson Hankemans et al., 2010). In the bigger picture, the technological development relating to, for example, smart-homes, smart-cars or the Internet-of-things is arguably driven by collective visions and myths about ubiquitous or pervasive computing (see Dourish and Bell, 2011). Therefore, it is important to be mindful with one's visions and concepts and to avoid making too stereotyped assumptions about people's life and needs.

The discussion in this paper will focus on embodied interaction. For decades, embodiment has been a central term in the phenomenological movement, positioned in opposition to Cartesian mind–body division, and has been adopted relatively recently into the discourse of cognitive sciences (Varela et al., 1991). The foundational value of embodiment as a ‘property of being manifest in and of the everyday world’ was acknowledged for HCI even more recently (Dourish, 1999, p. 2). However, in the HCI field, embodied interaction is often understood in the more narrow terms of body and bodily activity as means of interaction and control. Currently emerging technologies based, for example, on motion sensing and gesture recognition (e.g. Rautaray and Agrawal, 2015), gaze tracking (e.g. Lukander et al., 2013) or biometric sensing (e.g. Quek et al., 2013) are focused on utilizing bodily activity, gestures, and movement in the user interfaces of so-called smart applications. When looking at interaction through the concept of use, it is easy to see the physical movements having their instrumental value prioritized. Be it sweeping with a forefinger across the touch-screen of a smartphone, waving an arm in thin air or intently eying a smart coffee-machine, all that matters is how a user's action could be correctly interpreted by the application as an input. Such a narrow focus on movement or gestures through a priori instrumental value in user tasks (see, e.g. design guidelines in Microsoft, 2013) might hinder the formulation of the bigger picture regarding how devices with body-based input collide with and affect our lives through their intended embodiment. In this paper, we want to emphasize a broader understanding of the term embodied interaction, proposed as a comprehensive foundation for HCI that acknowledges the body through phenomenological and cultural accounts in addition to the technological and psychological ones (see Dourish, 2001).

As many of the significant embodied nuances and routines in our lives remain mostly ‘invisible’ and unattended by us, there clearly exists a need for new, comprehensive ways to become aware of and make explicit how the designs of new technological commodities collide with the flowing patterns of human movement. Consequently, interaction designers should also be provided with robust means to extend or turn their attention toward the more wide-ranging perception of the designs that affect people's embodied experience of their surroundings as a lived space.

1.1. Objective space vs. lived space

In investigating human interactivity with smart applications, one key factor is to understand environments as a spatial phenomenon. Phenomenologists (e.g. Husserl, 1970; Merleau-Ponty, 1962) have long stressed that the living human body is not a mere thing that rests in space like tables, stones or computers. Spatiality that we live in and through our bodies and around us is called lived space. This lived space differs from the schema of Euclidean 3D geometric space. Objective space can be scrutinized and measured with fixed metrics, and it has become the foundation of design in architecture, engineering, digital technology and the physical sciences. It is not only a useful tool, but objective space has dominated the understanding of space in Western culture since classical mechanics (Grosz, 1994; Husserl, 1970; Merleau-Ponty, 1962; Straus, 1966).

Lived space can be considered a much more fundamental notion of space than objective space because all living beings and the knowledge they possess depend on the sensual and intentional orientation to their environments. Even plants turn their bodies to the sunlight or push their roots into the soil toward water sources. The motion of plants may not be conscious or intentional (at least in the way these concepts are traditionally understood) but nonetheless plants have their own lived spaces. Of course, the lived space of humans is different and much more complex than plants. Merleau-Ponty (1962) has addressed how the lived space emerges reciprocally from the human body's perception and action, but it also carries complex meanings of social and affective relations that become enactively coupled with specific places and ways of interaction (Hutchins, 1995; Noë, 2009; Varela et al., 1991). Merleau-Ponty's (1962) notion of space (l'espace) relies on Heidegger's conception of being-in-the-world (In-der-Welt-sein) showing how fundamental the orientation of spatial living is for us. In fact, following Merleau-Ponty's thoughts, it can be argued that our postures and actions on the world, as constitutions of lived experience, are an integral part of our enactive consciousness, not necessarily being caused or driven by higher order propositional mental representations, such as plans of action. This notion relates to Suchman's (1987) seminal argument stating that the ‘planning model’ of action does not adequately describe situated activity of humans. Thus, movements of a living body are fundamentally intentional and mindful in themselves, in a manner that does not rely on reflective thinking but on recurring lived experiences of bodily action (Johnson, 2007; Varela et al., 1991). It is quite safe to say that the variety of everyday skills and coping with physical activities rely on such a body-based knowing originating in ‘muscle memory’ and which most of the time remains unattended by us (in pre-reflective consciousness). However, it is important to acknowledge that this ability is not detached from rational thinking, but contributes to and brings forth our thinking and our perception of a stable reality (Johnson, 2007; Varela et al., 1991).

Here, we want to promote the idea that technology should not be designed solely based on the notion of objective space, because it is after all only one modulation of space among many other equally coherent, equally possible formations. Objective space measures things in terms of their size, length or height, and their reciprocal spatial relations. This measuring effectively dismisses the subjective, bodily enacted experience of spatiality and the related kinesthetic perception of one's body (via body-image and body-schemas, Gallagher, 1986), and it does not appreciate the embodiment of intentionality toward the environment and its sensorimotor meanings and affordances (Gibson, 1979; Noë, 2004). Intentionality emphasizes the immediate relationship between the embodied mind and the lived environment, thus ‘in saying that the mind is intentional, phenomenologists imply that the mind is relational. ‘Being-in-the-world’ (Heidegger) and the ‘lived body-environment’ (Merleau-Ponty) are different ways of articulating this kind of relation’ (Thompson and Stapleton, 2009, p. 27).

This embodiment of intentionality should be seen as a fundamental source of affective, social, ethical or esthetic meanings we (as embodied subjects) naturally attribute to even the subtlest qualities of gestures and other body movements. Digital systems that respond to movement cannot understand the lived-through gestural meanings of the moving body, thus, ‘gesture recognition’, which has taken a pivotal role in HCI, is a misleading term. Digital devices can, for example, measure the coordinates of the body's movements in objective space and interpret them as pre-specified gestures or poses through mathematical algorithms and symbolic catalogs of movement information. Arbitrary mapping of these gestural objects to certain commands of a user interface may easily lead to unnatural or imposed movements that distort the flow of natural or habituated movements in social contexts. Let us imagine, for example, an augmented reality application that uses certain facial expressions, such as eye winking and smiling, as means to respectively browse and select visual menu options displayed in the augmented reality glasses. In public situations, it is easy to imagine social misapprehensions and potentially embarrassing situations this gestural control may cause, considering the importance of facial expressions in interpersonal communication of affect. Even if the social factor is ruled out, instrumental use of emotional expression, like smiling, would be ethically a very questionable strategy. Just because it forces you to ‘smile’ in order to fulfill some function, which in itself might have nothing to smile for.

From the perspective of lived space, body movement is not something that could be captured in mere physical dimensions and harnessed lightly by its instrumental value. Rather, it is fundamentally about how we—as intentional agents—feel, perceive, interact and communicate through our body in situated moments. It is something that essentially embodies and enacts what we are, what we know and what we experience (see Noë, 2009). In short, movement is about life. And that is something one should never handle without care.

1.2. Aim of the study

What is maybe not yet fully understood in the discourse of HCI is the dynamic continuum of human action(s) as an experiential whole. Instead of focusing on gestures as chunks of movement relating to users and devices, a choreographic approach (discussed in Parviainen et al., 2013a, 2013b; see also Loke and Reinhardt, 2012) implies putting the focus on the flow of movements we perform in our lived environments. Here, choreographies include all bodily movements, routines and activities in which movements appear to form meaningful interactions and relations in a lived space between different agents and inanimate objects. Unlike in the domain of dance, there is no choreographer who ‘alone could lead the dynamics of this constellation; rather, human and nonhuman agents have connections with each other that establish ongoing choreography’ (Parviainen, 2016, p. 62). In this process, technological objects can be conceived as so-called actants that influence human actions (see Latour, 1996), thus co-constituting choreographies through embodied practices and affordances being attributed to their design (as pre-choreographic elements). In general, this choreographic shift of focus helps to perceive (i) how discrete movements belong to a bigger whole of both spontaneous and orchestrated/arranged movements, (ii) how these choreographies differ across times and places, and maybe most importantly (iii) how movement and space are lived through by subjects (individual choreographies), and among subjects belonging a cultural community (social choreographies).

By shifting the focus toward choreographies in the lived space, we aim to unveil the multidimensional aspects and phenomena that reside within the HCI paradigm founded on embodied interaction. In particular, we are interested in the concept of embodied control constituted within the choreographies of interacting with technology. Instead of just seeing ourselves in control, it is time to also consider how these systems are controlling our own bodily movements and choreographies, for example, by expecting particular types of movements from us. We will analyze and describe the effects of some body-based interfaces in terms of embodied control in a manner that goes beyond the instrumental user-to-device control scheme. The following section first examines optional theoretical stances on the phenomena of embodied control, and then presents a recapitulated formulation of the concept in three dimensions that are instrumental, experiential and infrastructural. Finally, implications of the refined perspective are discussed in terms of interaction design and human–technology studies. Among the examples dealt with in the discussions, one particular domain of embodied interaction is more deeply covered. This particular example concerns interfaces for musical expression and interaction (in Section 3.1). Musical interfaces may not appear as representatives of mainstream user interfaces, but for the purposes of this paper, they are illustrative examples because of their orientation toward embodied activity (e.g. Jensenius, 2007) and fluidity of processes both in performing music and in transformation of musical practices (e.g. D'Arcangelo, 2004). It could even be argued that they anticipate aspects of rich multimodal and embodied user interfaces by providing ‘excellent examples of sensorially rich and temporally detailed human–machine interaction’ (Tzanetakis et al., 2013, p. 1119). All in all, however, we aim at illustrating the phenomena of embodied control that are not necessarily dependent on any particular technology or field of application.

2. DEFINING EMBODIED CONTROL

2.1. Instruments of control

Already at the dawn of the modern computer-era, Wiener (1954, pp. 24–36) identified the theory of control in terms of communicative messages (as acts of control) and the gained feedback response (relating to the effect of control). An early developer of embodied interfaces, Krueger (1977) followed similar line of thinking in his concept of a responsive environment, which refers to a system capable of perceiving human actions and providing action-relevant (auditory and visual) responses through the surrounding environment. Therefore, when thinking of the concept of embodied control, it seems straightforward to conceive of it through the idea of harnessing body movements as a means to produce input messages to a control surface in HCI. A marketing slogan for Kinect ‘You are the controller’ (Microsoft, 2010) summarizes this approach aptly: dedicated controller instruments are replaced with ‘you’, by treating your body movements (referred to as gestures) as instruments for controlling an interactive system.

Let us take a look at a currently prominent guide for designing gestures for embodied interfaces: Kinect for Windows: Human Interface Guidelines (Microsoft, 2013). What the guidelines in general emphasize is the ways that (i) gestures are appropriate for user task and its orientation and that (ii) the intended control gestures can be reliably recognized and distinguished. Several design principles relate to, for instance, physical and cognitive ergonomics and utilization of natural interactions. The guidelines are also aware that instrumental gestures may generate problems with different contexts of use, urging designers to take into account user orientation, mindsets, locations, expectations and sociological factors.

…Kinect for Windows sees a person holistically. Kinect for Windows users are constantly moving and interacting with the world, whether or not they intend to be interacting with your application. Your challenge is to detect intended gestures correctly and avoid detecting ‘false positive’ gestures. Keep in mind that users in a social context may alternate between interacting with the application and with other people beside them or in the vicinity. The application must be aware and considerate of the social context of the users. (Microsoft, 2013, p. 30)

Considering the compact and practical scope of the guidelines, it is understandable that the offered design principles work as general reminders of things to take into account, not trying to provide more specific criteria and guidance for creating appropriate designs, for example, from the point-of-view of different user orientations. Guidelines urge designers to acknowledge things that may affect the usability of the application. However, some big questions may be raised in the reader's mind: How to define the goodness of fit for gestures? How to define what is natural interaction? The guidelines avoid taking a stand on issues concerning how the application of gestural control can smoothly interface with the user's everyday movements. The focus is placed on detecting the instrumental movements relating to application use from the ‘mere’ natural movements, but the actual means to achieve this is left to the designers:

With Kinect for Windows, it's harder to distinguish between deliberate intent to engage and mere natural movement in front of the sensor. We leave the details of how to handle this up to you, because your specific scenario might have special requirements or sensitivity. (Microsoft, 2013, p. 89, emphasis added)

In the guidelines, gestures and their meaning are seen first through their instrumental value. Yet, it is important to notice that the basic significance of movements as choreographies of human life is somewhat recognized, but this aspect is mostly overlooked—or at least put into a secondary position.

In the literature of musicology and game studies, there exist two interestingly similar taxonomies of body movements relating to playing musical instruments (Jensenius, 2007, pp. 46–47) and playing games (Bianchi-Berthouze, 2013, pp. 49–51). Both of these taxonomies seem to make a basic division between movements instrumental to playing and other movements of the playing situation. The former class consists of required movements for control (sound-producing or task-control) but also movements that provide indirect support to performing them smoothly (ancillary, task-facilitating). The latter class includes movements that are not required but either accompany the playing (e.g. by following contours of musical elements or the player's role in a gameplay) or otherwise relate to situated expression and communication. Although these taxonomies explicitly take into account non-instrumental movements, their starting point nonetheless lies around the instrumentation of control, against which movements are defined and ordered. The players’ orientation is also a matter that needs to be accounted for. For example, affordances of action are not necessarily constituted within the organization of an instrument's mechanical functionality (e.g. gameplay mechanics). As shown in the example provided by Bianchi-Berthouze (2013, p. 48–49), expert gamers sometimes wanted to perform sport-like movements (involving the body in a simulated manner) in playing Nintendo Wii sport games, even though they knew that, due to the accelerometer-based controller, they would perform better in the game by moving the game controller with tiny and efficient hand movements (with little resemblance to the sport simulated in the game scenario). Thus, two distinct player orientations were outlined in the example: one of them focusing on mastering the instrument efficiently, while the other implies an engagement to the game scenario through its action affordances and the related choreographic engagement.¹ If we chose to prioritize the latter orientation, we might actually come up with different taxonomical organizations where ‘pure’ instrumental actions (relating to the actual control mechanics) become subordinate chunks of movement belonging to more holistic, goal-directed continuums of movement (e.g. emphasizing a ‘golf-swing’ action over separately timed instrumental movements of a controller).²

As we can see, taking care of different user orientations matters. But in order to actually perceive and outline these orientations, the researcher ultimately needs to resort to a subjective stance on one's space and the ways the instrument becomes part of it.

2.2. Control we live by

It will help our analysis on embodied control if we first follow the contrast between objective and lived space—the two basic phenomenological categories of space we presented earlier. In other words, we should first consider the differences in seeing environments in terms of how they are lived through their indwellers instead of just treating actions and spatiality in objective and functional terms. Due to the nature of computing, smart technologies ‘sense’ their environment through measurements and formal rules and treat human bodies like they were any objects that could be mechanically measured and modeled. While, for example, motion-capture systems can detect, measure and categorize movement of such objects, in a lived space we interact within, our actions in and responses to these systems are not mechanical but unavoidably intentional and colored with affect—in a manner our actions on the world always are.³ Within lived spaces, the distinction between ‘natural’ movements and instrumental movements (relating to intended user functions) becomes increasingly hard to identify and even impractical because of the necessary overlap between the two. Instrumental movements can never be ‘empty frames’ without some natural, experiential meanings tied to the movement qualities. This was indicated, for example, in the Wii-game scenario presented above, where gamers used more involving movements for controlling because of the fun they provided (Bianchi-Berthouze, 2013). On the other hand, one does not necessarily have to experience instrumental movements as being directed to a system. As computing is becoming ever more ubiquitous and embodied, we are no longer mere users controlling systems from the outside, but rather, we dwell in these systems (Parviainen, 2011).

So, how would an experiential conception of embodied control differ from instrumental control? The difference comes from the perspective: While instrumental control is about providing technical means to control—in terms that designers would describe as being convenient for a user task (in the third-person)—experiential control is about how individuals feel or conceive control in the first-person, in lived moments, in the lived space, through their body awareness. In other words, it is about how designs co-constitute experiences of control within interaction, not necessarily having any a priori assumptions about the object of being controlled. As we saw in the Wii-game scenario, the players’ orientations changed the experiential aspects of control: while the expert players prioritized mastering of the game controls, they were occasionally willing to give away that sense of control for an opportunity to experience the game differently. Therefore, experiential control is also a question of how we are orientated to our lived space (including also interpersonal factors), what kind of action affordances we tacitly sense that either support or guide and restrict our momentary intentions and the related bodily engagement. For an expert player wanting to master the game as a system, the idea of performing sport-like choreographies may appear as a restriction or unnecessary imposition (i.e. choreographies are ‘pushed’ at the player). By contrast, for a player wanting to engage with the game scenario, performing these choreographies provides support for his or her intentions (i.e. choreographies ‘pull’ the player into a game).

The concept of experiential control may be seen in a 2-fold manner: first, it is about the feeling of being in control within a perceived reality of a lived situation. Indeed, controllability has been defined as a key quality of a usable application in the HCI literature (Alonso-Ríos et al., 2010) and as the standardization system of ergonomics for HCI (ISO 9241-110, 2006). In the past, control was ideally implemented with dedicated controllers, but contemporary visions of smart-homes strive to make computing ‘disappear’, providing users with the ability to meld the technology into their everyday lives (Harper, 2003, p. 24). In the course of making technology invisible, one trend is to move away from dedicated controls to smart, proactive and automated systems that follow and learn the habits of the residents and use this surveillance data for anticipating and automatically responding to their routines (see, e.g. Aldrich, 2003). Although many might be fond of the central locking of doors and windows, or lights coming on when someone enters room, it might also severely disrupt the feeling of being the one in control. Another possible trend in making controls invisible is the so-called enactive approach to user interfaces (see Froese et al., 2012). The basic idea is that control should be implemented as experientially transparent by avoiding positioning it between the user and his or her operational environment. For instance, instead of using hand gestures for giving dedicated commands to the system, the device or its functions can themselves experientially become an extension of our hand, seamlessly augmenting the enactive sense-making and control of the lived space. The enactive approach strongly relates to Heidegger's (1962) phenomenology of tool use which concerns how objects separate from the body can experientially become extensions of our body,⁴ fused into an extension of a lived space and the related body-image and sense-making. It is indeed important to acknowledge the transformative effects of tool use (with regard to enactive perceiving of the lived space) that force one to consider that ‘many different ways of perceiving the world are possible’ (Froese et al., 2012, p. 373), and that in this manner lived space also extends to virtual spaces experienced in an engagement with technology (such as user-interface widgets, computer games or social-media applications). Tool use also reminds us that experiences of control are something that does not always require conscious attention to objects being used for controlling or interacting. For example, a wearable musical instrument, as a device that maps movements to musical parameters, can easily become transparent at a point where the player starts to feel that his/her body actions directly transform into musical interaction.

The second aspect of our account of experiential control is about the conflicting feeling of having to self-control or regulate one's own (intended or habituated) movement and actions in a present lived space. There are plenty of possible situational factors relating to implementations of technologies that might contribute to the need felt for modifying or restricting our movement, or regulating our spontaneous bodily display of affect. Such effects on movement are induced, for example, by needs to conform one's bodily actions to physical barriers, to social norms of an environment or to certain behavior expected by smart systems. For instance, future smart-home systems might make use of gestural interfaces based on optical motion-capture technology (similar to Kinect). Although the implementation of such interfaces is physically invisible, the active field of motion capture is fully analogous to any physical barriers or social expectations that shape our movements at home. When such a system continuously monitors and traces movement in a living room, potentially reacting to the specific movements (gestures) in space, people have to be careful in their moving and posturing, to avoid giving ‘wrong’ signals through their body. They might even learn to intuitively suppress their activities to avoid performing certain kinds of bodily movements in certain places, eventually constituting new or modified norms of bodily behavior. Gaze-tracking glasses, as a means to interact with smart household items, might pose similar issues of restricting the experiential freedom of looking around. In public situations, there is often an increased pressure to conform to social and cultural norms in the ways one interacts with technology (e.g. Reis et al., 2008). For example, we might be embarrassed to use gestures or voice because of the deeply rooted communicative (interpersonal) expectations relating to these interaction modalities. This is especially true when instrumental gestures clash with the natural affective communication among other people.

It is also important to note that the two mentioned aspects of experiential control do not exclude each other. For instance, while the wearable musical instrument of the previous example might have the great and natural feel of being in control of musical sounds (via transparent control within free body gestures), it might make you feel embarrassed or uncomfortable with the movements and poses emerging through the playing performance. Mapping free body or limb movements to musical expression, by using optical motion capture or sensors such as accelerometers or gyro meters, might also severely restrict the player's freedom to spontaneously perform body movements and expressions that are not directly related to sound-producing (e.g. ancillary, sound-accompanying and communicative movements, see Jensenius, 2007, pp. 46–47). Therefore, depending on the orientational disposition of a player, experience of playing such an instrument can appear as a feeling either of gaining or losing control.

2.3. Infrastructures of control

Dourish and Bell (2007) have investigated the conditions for ubiquitous computing by considering spaces as infrastructures and focusing on embodied action in both practical and cultural organizing of such infrastructural spaces. Their approach to the organized utilization of spaces is applicable to the analysis of embodied control. While zooming out from the experiential focus outlined above, we build the third dimension of infrastructural control roughly on these ideas. This dimension basically concerns the question of how every piece of technological design ultimately participates in establishing and organizing infrastructures that control people's movements and embodied use of space. From the digital alarm clock that wakes us to the fully automated and integrated smart-home system, computer code is extensively and intimately woven into the fabric of our everyday lives (Kitchin and Dodge, 2013). In fact, all design choices with regard to built environments partake in this phenomenon. However, infrastructural control is not only in the hands of designers, but rather, the creation of these infrastructures is an emergent process involving larger groups of people. Nonetheless, every design contributes more or less to this societal impact.

Before we get involved further with the question of infrastructural control, it is necessary first to consider how technological infrastructures contribute to our perception or how perception is technologically mediated. McLuhan (1964) is well known for his notion of technology whereby technological devices are an extension of the human cognition and organism. Extensions of the body cover a wide variety of different instruments: from handmade tools to vehicles and technologies for transportation and media. Infrastructures, such as roads and lighting, and other fixed structures, such as buildings, can be seen as extensions of the human organism. While these extensions open up possibilities, that is, work as affordances, they simultaneously impose constraints and frames on human behavior. Ihde (1990) in his philosophy of technology goes even further and argues that technology can never be separated from our perception and experience. Technological items seem to become part of our embodiment in the manner that technology mediates our experience of environment or world. Ihde calls relations between human beings and technology embodiment relations. In an embodied relation, we experience the world and objects through a machine, so in this sense, technical devices and their infrastructures always transform our perceptual capacities. In activity theory (see, e.g. Kaptelinin and Nardi, 2006), the mediating role of tools is stressed. In its emphasis on social structures, activity theory sees the role of tools as 2-fold. First, the tools shape the ways in which we interact with our environments. Second, since tools have been created and transformed in the development of a given activity, they themselves reflect the knowledge of people who have been involved in it. In other words, in the use of tools social knowledge is cumulated.

While strongly relating to technological and material organization of lived spaces, the heart of infrastructural control lies in immaterial interpersonal phenomena such as construction of social and cultural norms and habits; for example, how people are expected to organize the furniture in living rooms, or what kind of technology should be included. Thus, each culture's present conceptualizations of what is home, how it is normally organized, consequently creates models for what kind of movement, choreographies and bodily engagements with the lived space are normally expected. For example, we currently might be on the verge of an emerging norm where almost every family member is recurrently or continuously engaged with smartphones and other mobile devices, and day-to-day communication with family and friends also involves the use of these gadgets (e.g. Lin and Atkin, 2014). According to a marketing survey from the Sparkler Research Agency (2013), 30% of families in UK regularly use smartphones or tablets to talk to each other when they are in the same house. Such a change in family communication would also involve changes in the lived space families reside in. These might include, for instance, a shift of emphasis in communication toward favoring technologically extended perception of the social space. Another home-relevant example could consider the camera-based motion-capture technologies, which may currently be restricted to the use of gaming systems (such as Xbox One). Therefore, the effect that these technologies have on infrastructures of the home might now seem trivial. But as discussed in the earlier subsection, even the current implementations carry potential for significantly changing the organization of a lived space. Moreover, it is rather safe to assume that the volume of applying similar motion-tracking technology, for example to different smart-home applications, will increase in the future. At that point, emphasis on designing gestural controls that do not inappropriately collide with people's daily lives becomes a much more serious issue.

Besides being manifested through sociocultural habituations, infrastructural control can also be based on and be enforced by institutional norms, guidelines and legislations. For example, there are usually different kinds of regulations with regard to building houses and technology. Also, big manufacturers of technological products (such as Apple, Microsoft or Samsung) may have the power to directly influence trends and emerging norms through their products and marketing. One part of the infrastructural control in the hands of institutions relates to opportunities to monitor people's movements and activities through specific technologies in specific actual and virtual milieus. A large part of such monitoring may just locally serve the appropriate operations of certain smart applications, but there is always the possibility that the data from such surveillance are stored in a concentrated repository for whatever future purposes of a corporate or governmental institution. One can also see monitoring in the light of constituting and upholding infrastructures of embodied control. For instance, Parviainen (2016) has concluded in her account of choreographies of biomonitoring that knowledge infrastructures of monitoring (involving Big Data generation) are reciprocally connected to personal embodied practices.

What are then the differences of infrastructural control when compared to the other two dimensions of embodied control? Infrastructural control does not concern the questions of how body movements can be dedicated to controlling something, but it concerns how infrastructures of devices and the related practices and social norms control our movements and our embodied use of a lived space. Also, it does not concern particular experiences of control in lived spaces, but construction of infrastructures that control and organize our lived spaces and the perceived action possibilities within (i.e. through affordances that either allow or constrain). There is, however, one relevant link to instrumental control: it relates to the ways that designers can make dedicated choices in their designs that either support or suppress the construction of certain infrastructures that have an effect on movement.

2.4. Discussion of embodied control

We have performed an analysis on the concept of embodied control in order to reveal phenomena relevant to interaction design that go beyond the idea of harnessing body movements as instruments of control. This idea might usually be the first and also the most straightforward way to conceive embodied control, for example, in cases of providing gestural control for applications. In all, three dimensions of embodied control (instrumental, experiential and infrastructural) were identified and they are summarized in Table 1.

Table 1.

Three dimensions of embodied control and their relevance to interaction design.

Instrumental control	How designs harness body gestures and movements as instruments in control surface in HCI
Experiential control	How designs participate in constituting a feeling of being controlled and a feeling of being in control Push effects: contribute to affordances that guide or constrain tacit orientation and bodily engagement Pull effects: contribute to affordances that support or facilitate tacit orientation and spontaneous bodily engagement
Infrastructural control	How designs participate in establishing and modifying both material and immaterial (i.e. sociocultural) infrastructures that control the everyday embodied use of lived space

With regard to the lived experience of control, we find it useful to investigate the relevance of this dimension to interaction design through push and pull effects. These effects concern situational factors (relevant to design) that either push forth a perception of barriers that we feel guiding or forcing us, or support our orientation and thus adapt to our pull. Push effects in design relate to a felt conflict between our tacit orientation and the action affordances of a built environment. It should be remembered that affordances here can also function as constraints: that is, they imply what not to do or what you might need to do instead. Pull effects are the opposite as they relate to a downstream perception of action affordances that corresponds to the situational intent and our tacit orientation of bodily engagement. Subjectively, they seem to enable and encourage ‘just the right kind’ of engagement with the design. In simpler (maybe even oversimplified) terms, one could say that push and pull effects relate to the feeling of being controlled and to the feeling of being in control, respectively.

Let us imagine, for example, the task of transforming a normal city street into a so-called people's street, which will have restrictions on car-traffic while giving more freedom to pedestrians. In order to enforce such a transformation, one might want to design new constructions for the street that give clues to both car drivers and pedestrians that the environment and the expected choreographies differ from the normal streets. These constructions may include, for example, plantings, artwork or park benches placed to block straight driving lines (push effect for cars), while also encouraging pedestrians to ‘take over’ the street space (pull effect for people).

Of course, it is often far from easy to be absolutely sure what aspects of the design concern push or pull effects, since it is fundamentally a matter of the individual's situated experience. Moreover, it is important to remember that push and pull effects also relate to those situational factors that are outside the domain of the particular design and thus cannot always be influenced by it. With regard to designing embodied interactions, one may well argue that environments or systems that are designed to use movements non-intrusively, embedded in the meaningful movements of the context (as a whole), might more easily contribute to the rise of pull-type affordances (and immersive tool-use experiences). But on the other hand, it could as well be argued that harnessing natural movements to embodied control might lead us to being more self-critical and self-sensitive toward our normal movements in that given space, thus creating push effects. In some contexts, a deliberately clear distinction between the functional movements and the movements natural to the context might generate a desirable feeling of being in control, as also implied in the Kinect guidelines (Microsoft, 2013).

An interesting case to be investigated in terms of pull and push effects is tangible user interfaces (TUIs; term and abbreviation coined by Ishii and Ullmer, 1997). TUIs can be argued to be a reinvention of classical tools. Once you have a, say, hammer in your hand, is the design more an enabler (pull effect) or does it merely force you to hit a nail in a strictly designed way (push effect)? It appears that in the recent enthusiasm for TUIs, the control-aspect has not been adequately elaborated (see, e.g. Djajadiningrat et al., 2004; Klemmer et al., 2006; Müller et al., 2014). In this kind of notions, the concept of control is only used to refer to the user-interface elements, which enable the control of a given application. What would be needed is a better understanding of the importance of perception of being in control.

Another issue closely relevant to pull and push effects lies in the conception of persuasive technologies. As the name suggests, this paradigm deliberately aims at persuading people or users to change their behavior through engagement with a piece of technology. Persuasive designs try to outline and utilize certain motivational affordances (see Hamari et al., 2014), which increasingly relate to gamified interaction and situated psychological rewarding. A known example of persuasive design is Piano Staircase (discussed in Peeters et al., 2013) in which a staircase next to an escalator was transformed into a functioning piano keyboard, with the intention to draw people toward using the stairs. Another example of persuasive technologies is provided in car dashboards with gamified rewarding for eco-friendly driving. These dashboards, for example, show visualizations of leaves and vines that grow and flourish in accordance with the efficiency of driving (Froehlich, 2014, pp. 578–579). Finally, the recent Pokémon GO phenomenon (see, e.g. Greiwe, 2016) counts as an effective example of creating an augmented reality game world (through smartphones) that gets people moving and exploring their surroundings in a new way. Although the main ethos in using these kinds of motivational affordances leans toward pull effects, it seems that persuasion can potentially relate to both push and pull types of affordances. Arguably, the perspectives of embodied control presented should also be beneficial for the investigations of persuasive or gamified interaction design.

One may also ask whether push effects are negative and pull effects positive. Such values may be attached to these concepts, but nevertheless, it is not that straightforward at all and will ultimately be dependent on the contextual whole. For example, highway speed cameras are designed to guide us to restrict our driving speeds, possibly against our momentary will. In a broader sense, speed cameras belong to the infrastructure that upholds traffic rules and the norms of safe driving. Therefore, the potentially experienced push effects constituted by speed cameras (due to constraining our driving) may have both positive and negative values, depending on the viewpoint. In all, it should also be noted that none of the dimensions of embodied control are either good or bad in themselves. Either positive or negative aspects can be attributed to any of them.

3. DESIGN IMPLICATIONS

In this section, we further discuss how the conceptualization of embodied control and push/pull affordances introduced above has an effect in designing and pre-choreographing interactive systems and interfaces. First, the issues of instrumental and experiential control are discussed within a context of musical interfaces. Then, we switch to a wider sociocultural perspective in order to reflect matters relating to infrastructural control. Finally, we summarize the discussion of implications by considering design perspectives and methods.

3.1. Instrumental and experiential control in new musical instruments

Musical interfaces have already been considered earlier in this paper, but now let us take a slightly closer look at these systems of interactive computing, also known as New Interfaces for Musical Expression (NIME).⁵ Following the HCI tradition of conceptualizing interaction between humans and devices as a composition of inputs and outputs (e.g. Card et al., 1983, p. 26), NIMEs too are often conceived as information systems consisting of control interfaces (for input) and a sound generation unit (for output). This is evident, for example, in the digital musical instrument model provided by Miranda and Wanderley (2006, p. 3), in which gestural controller input and its mappings to sound production are considered. While such a conceptualization gives a clear picture of how technical instrumentation and the flow of information is organized, it does not provide support for understanding the experiential domain of musical interaction, in which embodied intentionality of actions and sensorimotor integration are central concepts (see Tuuri et al., 2009).

NIMEs usually utilize wide varieties of different sensor technologies and interaction paradigms in creating a diversity of novel control spaces for musical interaction (Lyons and Fels, 2014, slides 17–20). We want to stress here the importance of looking beyond the input–output model, thus not conceiving of control spaces merely as instrumental spaces delineated by sensors, physical performance and the related input information. In order to better understand the musical interaction in these spaces and the ways that embodied control appears in them, it is helpful for us to switch the focus away from technologies and to try to see such a control space from the subjective and intentional viewpoint of the musician playing the instrument. That is, how the player is oriented experientially to the lived space of musical engagement. Such an orientational disposition is essential in delineating how experiential control (and its push and pull effects) appears, and what kinds of choreographies are afforded or expected within a given stance to musical interaction.

First, one can outline an orientation in many ways comparable to playing traditional instruments such as the piano. In this stance, the actions on a control space are primarily for producing (and precisely controlling) musically organized sounds. Hence, the related NIMEs enable a practical, ergonomic and preferably tangible interface, which is mapped for making music in terms of musical concepts, structures and habits (i.e. the musical potential is seen through producing, for example, notes, scales and/or chords). We can call this a musicking stance.⁶ NIMEs that strongly correspond to this stance include, for example, touch-based, tangible instruments like the Continuum fingerboard (Haken et al., 1992) or a music glove with discrete touch sensors for each note (Myllykoski et al., 2015). Second, a clearly contrasting orientation could be called an indwelling stance, in which a person explores ‘reversibility between sounds and movements that is grounded in imagery or real events in the Lifeworld’ (Parviainen, 2011, p. 645). Lifeworld here refers to epistemological relatedness between the embodied mind and the environment, co-constituted by the player's actions. Movements and gestures within a control space, and their sonic mappings, thus give rise to a subjective sonic world for one to dwell in and interact with, while being analogous to sensorimotor interaction with the physical environment—its ecological consistency and its ‘resistances’ (see, e.g. Peters, 2013). According to Parviainen (2011), indwelling is closely related to building bodily knowledge through both passive and active movements in the sonic environment. Many NIMEs that map sounds to free body and limb motions correspond well to this stance, for instance the Embodied Generative Music system (utilizing full body motion tracking, Eckel and Pirrò, 2009) or gesture-based music gloves (utilizing wearable motion sensors, Mitchell et al., 2012).

In summary, the musicking stance is in general oriented to the instrument and its affordances for making music while indwelling concerns being ‘inside’ the instrument and its realm. To some degree, it may also be argued that the experiential aspects of musicking and indwelling are both potentially present in playing, regardless of the given NIME and its features. In this sense, the nature of stances could be compared to listening modes (see Tuuri and Eerola, 2012), which refer to different intentional constituents of meaning-making in the process of listening. Although the musicking stance might be encouraged in playing instruments where controls are laid out in musically structured ways, it does not prevent a player from immersing themself into an existence within a sonic realm (in playing) that is not necessarily founded on the given musical structuring. Conversely, instruments providing more environmental or ‘ambient’ action-sound affordances (implying support for the indwelling stance) may as well be used in expressing with musically organized rhythms, pitches, harmonies or clusters of sound. Finally, it is important to remember that while the musicking stance is oriented to the instrument, the intentionality of musical interaction yet extends beyond technical instrumentation. For a skilled musician, any instrument can become functionally transparent, meaning that the player feels as if he or she were directly interacting with music (Nijs et al., 2013). Hence, the musician's intentionality is directed toward objects of music, which can be considered as ‘inner’ models of music (as suggested in Nijs et al., 2013) or as metaphors and image schemas relating to embodied conceptualization of the music being played (Johnson, 2007; Wilkie et al., 2013).

By introducing the two stances we want to point out that actions and movements of musical interaction relate to orientational dispositions that provide different frames for understanding the intentionality in bodily musical interaction, including the ways that situated affordances and push or pull effects of embodied control potentially become organized in one's experience. When considering the engagement with a NIME from the musicking stance, pull effects may, for example, relate to the ease and robustness of conceiving action possibilities for playing specific notes and melodies, thus controlling the sonic output in terms of affordances relating to objects of musical discipline. From the indwelling stance also, pull effects are about the feeling of being in control—albeit in a very different way. Arguably, pull effects in this stance relate to the qualities of NIME to allure the player into trying out, exploring and learning the virtual sonic world (and its consistent ‘sonic ecosystem’ within). In a way, this stance may be called existential, because the alluring qualities of sensorimotor coupling of sound and movement can make the interaction and the related movements themselves rewarding or pleasurable, even without any clear goal to play specific premeditated sounds. Therefore, the felt agency within an action-sound ecosystem can be considered as a motivational (pull) factor in itself.

In general, it can be said that push effects within both of the stances should relate to the felt intrusiveness of choreographed movements required in performing. Basically, this is a question of how much these (often implicit) pre-choreographies attributed to a design cause conflict with the experience of freedom or spontaneity in the flow of musical expression. NIME pre-choreographies relevant to the musicking stance habitually resemble the ways traditional instruments are played, in a sense that they force one's hand or both hands (or mouth) into specific playing positions and actions (‘glued’ to the instrument), but otherwise permit free movement (within the limits of the mobility of the instrument). Push effects may also relate to the incoherent mapping strategies with respect to musical control (that is, how musically inconsistent the mapping of physical actions and sounds is). The indwelling stance again is different, for example in respect of permitting freedom of movement. In a way, the instrumental mapping of movements in NIMEs most relevant to this stance focuses particularly on utilizing our very ‘natural’ movements. If we manage to immerse ourselves in sensorimotor dwelling within the sonic feedback of the system, we might not experience any intrusive push effects. However, playing with the movements of your whole body inevitably leads to extensive, obtrusive, even dancing-style choreographies and poses. In some systems, even the spatial position of the player (e.g. on the stage) can be mapped to some controllable sonic parameters. Thus, embodied control through pre-choreographies is all-invasive and potentially very intrusive.

Again, this is not to say that choreographic control, manifested through the design, in itself would be either good or bad. The appropriateness and value of pre-choreographies of a NIME designs depend heavily on the context and the purpose of movements. During a stage performance, tightly choreographed and imposing playing movements can be very appropriate, communicative and even inspiring to the performer and the audience. On the other hand, as a practical tool of a musician, having to perform dance-like choreographies might not work that well. But if considered as a therapist's tool, such choreographic control by means of alluring—or even pushing—a person into producing certain movements can be very beneficial (see, e.g. Bergsland and Wechsler, 2015; Vogt et al., 2009). In such contexts, the sonification of movements builds up motivational affordances, through which a therapist can choreograph certain patterns of movement (for a therapeutic purpose) to be performed. It has been found that felt musical agency in movement can also reduce the exertion in a strenuous physical activity (Fritz et al., 2013), further illustrating the rewarding effect of an agency-relevant sonic feedback.

There currently exist quite a few theoretical accounts that describe NIME-related design spaces. These concern, for example, functional characteristics of different instruments (Birnbaum et al., 2005), dimensions that describe movement in terms of range, precision and haptic feedback (Bongers, 2006, pp. 157–175), or principles for developing interface technologies for expressive musical instruments (Overholt, 2009). All of these accounts offer a valuable contribution to outlining a conceptual dimension space of possible NIME design attributes, but their scope of analysis does not specifically embrace experiential stances and intentionality of interaction, even though many of their dimensions are clearly relevant to the experiential perspective. In general, the focal point of these accounts is more or less linked to the instrumental (i.e. input–output related) conceptualization of interaction. This is also reflected in the taxonomy of functional aspects of musicians’ movements (Jensenius, 2007, pp. 46–58), which is inspired by the concept of instrumental gesture (see Cadoz, 1994). In the taxonomy, movements that have an effect on sound output are put in focus, and only them are treated as goal-directed actions. This may encourage overlooking the goal-oriented nature of embodied interaction that extends beyond the instrumental functionality (i.e. activity directed to more holistic musical objects/goals, such as patterns of melodic progression and phrasing, that can also be manifested in ‘non-instrumental’ movements). Moreover, with regard to the functions of gestures, Cadoz (1994) has also formulated a notion of an epistemic function, which we interpret as referring to knowledge and information about the environment being acquired (and enacted) through movements. Unfortunately, such a function has not been included in the account of Jensenius, although it seems that it would fruitfully tap into experiential (sense-making) aspects of interaction, especially in relation to the indwelling stance.

In the development of NIME theories, there is room for improvement in terms of describing how a design space (or its dimension) should account for lived experiences of embodied musical interaction. Within an experiential perspective, the affordances and goal-directedness of musical interaction can be re-considered through different orientational dispositions (such as musicking, indwelling and those yet to be discovered). Also, instead of focusing too much on the instrumental scope of controlling sound, more effort could be put into acknowledging the reciprocal relationship of control between player-actor(s) and instrument(s). We feel that one way to reveal aspects of this reciprocal relationship is to consider musicians’ movements as choreographies and to evaluate what kind of choreographic elements (i.e. pre-choreographies) are attributed to NIME designs. Within a ‘choreographic dimension space’, it would not only be important what kind of choreographies produce certain sounds, but also what kind of choreographies are constituted and expected in playing, and what kind of experiences of control (push and pull effects) they induce in their contexts.

Through the discussion on NIMEs, we have tried to exemplify how the perception of push and pull effects depends on different subjective orientations. That is, how a person engages with the NIME in lived terms, what kind of intentional aspects are constituted in the engagement and how he/she conceives of the musical interaction and its affordances. We think that the orientations discussed here have relevance to embodied interaction that extends beyond music-related applications. Earlier in this paper, comparable orientations have already been discussed in the context of gameplay interaction (see Section 2). Of the two orientations, to some degree, the musicking stance corresponds with the traditional idea of a task-oriented user who sees himself/herself operating the device from the outside, and who seeks convenient and discrete functionality for accomplishing the intended goals. From this stance, choreographing of natural movements may feel intrusive and thus they may better be considered as clearly separated from functional movements of an emblematic nature. The indwelling stance is quite the opposite. In this stance, a person positions himself/herself inside the system, where he or she is not necessarily driven by any clear task, but rather, by motivations arising from being and interacting with this Lifeworld of augmented reality that combines actions with sounds. This seems to correspond somewhat with ideas of invisible and ubiquitous computing, where technology more or less transparently embeds in our natural embodied being and interaction with the environment (and the emerging affordances).

3.2. Control infrastructures of technology, information and human action

As bodily beings, we are always part of some technological ensembles and their infrastructural control. What constitute infrastructural control is not just the technological gadgets, as these are simply material components, but also the entire set of related practices and norms around them and the technological needs and interests they serve (Guattari 1995, p. 36; Mumford 1995, p. 305). Many current visions of future technologies suggest that standardized technological ensembles such as the Internet-of-things will be taking place on an ever greater scale. In these visions, computerized everyday commodities such as refrigerators, ovens, cars, mobile phones and toasters start to communicate with each other, forming larger assemblages in which human activity, information and materiality intertwine (see, e.g. Savat, 2013). While aiming to offer everyday convenience, well-being, entertainment or security for their users, these technological ensembles are also increasingly sensing us and gathering information for constructing data profiles of individuals and groups. Systems are becoming increasingly proactive and thus reliant on different types of user-data, data-driven models and algorithms to anticipate certain actions and predict their likely consequences with a view to eschew risk and forestall unwanted actions.

Proactive computing as a form of control curtails a person's range of future options, to allow or disallow a person to act in a certain way. But this kind of control does not just rely on immaterial (e.g. social) practices and norms, but on algorithmic infrastructures as well—the ‘smart fabric’ (Kûchler, 2008) into which we ‘code’ our lives through quantified choreographies among devices. This is the crucial mechanism of the assemblage that determines proactive operations of a system and algorithmically enforces the control. But who is in control, after all? As our actions are involved in the code generation, one can well argue that the resulting control infrastructure is (at least partly) based on our own subjective expressions. How much the user feels he or she has control over, for example, proactive functions of smart-home systems, depends on many aspects. One of them is the procedural transparency, the system either offers or does not offer to users with regard to the ways it gathers data from their actions and how it is utilized. The idea is often promoted that algorithmic processes and data-driven systems are purged of human bias and based on neutral and objective decisions (Ajana, 2015), but the algorithms ultimately reflect specific purposes of applications, together with the explicit and implicit values and intentions of their designers. Choices about data, connections, inferences, interpretation, thresholds for inclusion (in classification) and the overall procedural transparency to the users are neither neutral nor objective.

One simple example to demonstrate the relationship between human activity and proactive applications is playlist-based music listening (see, e.g. Hogan, 2015), where playlists are curated according to certain criteria (e.g. mood) and statistical data and models. Contextual factors (such as location, time and biometric measures) can also be used to enhance the situational appropriateness (e.g. Wang et al., 2012). Music listening is usually considered as a free form of cultural participation where choices of music reflect individual experiences, desires and identity as well as cultural identity and norms. Although listening at first maybe does not seem to have anything in common with choreographies, playlists, as chronologically organized choices of music, can also be seen in terms of choreographed activity. Thus, curated playlists—based on either human curators or algorithms—functions as an instrument to choreograph personal listening procedures. Without adopting any opinion on the appropriateness of the curated playlists from the listener, they ultimately require a certain givenness to algorithmic control. In terms of infrastructural control, it is not just a question of playlists and their algorithmic underpinnings, but rather how new practices and cultural norms emerge in relation to the general acceptability and desirability of ‘giving away’ some of the intimate relationship with music. Should the automated playlists become more common, they potentially start to have impact on the norms to which we conform our habits—how we are involved with music and how we participate in music as a culture. Although we may partake in shaping the algorithms, ultimately it may as well be that they are shaping us (Steiner, 2012).

Ajana (2015) suggests that the proliferation of data and profiles across networks and platforms gives the impression that identity is increasingly ‘abstracted’ from the lived bodies in a way that emphasizes a kind of Cartesian approach to body and mind. A Big Data techniques approach sometimes reduces individuals to what Deleuze (1992) calls ‘dividuals’; bits and digits dispersed across a multitude of databases and networks, and identified by their profiles, tags, pins, tokens, credit scoring and so on, rather than their subjectivities. They do not address people as whole embodied persons with a coherent, situated self and a biography, but rather make decisions on the bases of singular signs (Ajana, 2015). This, in turn, poses many ethical challenges in terms of the ways in which practices of collecting data and profiling mechanisms end up partaking of processes that impose certain identities while obstructing others.

In Value Sensitive Design, Friedman et al. (2006) demonstrate one attempt to provide an ethical framework that accounts for human values in a principled and comprehensive manner throughout the design process. Their list of human values comprises human welfare, ownership and property, privacy, freedom from bias, trust, autonomy, identity and environmental sustainability. Values are viewed neither as inscribed into technology, nor as simply transmitted by social forces. This comes close to Roger's (2006) critical view of Weiser's vision of calm computing. Rogers suggests that instead of trying to design ‘calm’ technologies, with an intention to embed them into our everyday life in a natural and fitting manner, we should also design them to be exciting, stimulating and even provocative—causing us to reflect upon and think about our interactions with them.

3.3. Design perspectives and methods

The contribution of this paper for designers lies in reformulating the concept of embodied control in a manner that gives rise to novel design perspectives. It encourages problematizing some established perspectives that may often be taken as evident truths. Most prominently, as already discussed in the paper, these include the oversimplified schema of taking the concept of control in human–technology interaction predominantly as something that the user directs to the device he or she wants to operate, as well as seeing gestural and body-related input methods as self-evidently easier or more ‘natural’ ways to give control to the user. Our analysis of embodied control elicits the omni-directionality of control between the user and the device, and within the assemblages of human agents, devices and material and social environments. On the other hand, it also emphasizes the perception of control as lived experience involving push and pull types of affordances of the lived environment. But most importantly, it helps make more explicit the embodied control over people that designers have in their possession through the designs. Table 2 summarizes the above discussion into design perspectives in accordance with each type of embodied control.

Table 2.

Summary of design perspectives relating to the dimensions of embodied control.

	Focal point of design	Philosophical position
Instrumental control	Just as the designed instrument of embodied interaction provides control to the user, inevitably the same instrument enforces embodied control on user(s). The embodied control on the user may be outlined as pre-choreographies of physical (or imaginary) movement becoming attributed to design. This choreographed attribute of control may be either unintentional (even unnoticed) or it can be intentionally utilized (e.g. as in therapy)	– Objective third-person perspective on agency and context – Emphasis on information technology and psychophysiology
Experiential control	How the designed setting of embodied interaction constitutes push and pull effects in the user's experience of control. In other words, how different aspects (such as the pre-choreographies) of the design contextually contribute to (i) the feeling of being forced or disabled, and to (ii) the feeling of being enabled and in control. Push and pull effects of design relate to the user's orientation and the ways he or she perceives contextual action affordances	– Lived-through first-person perspective on agency and context – Emphasis on phenomenology and ecological perception
Infrastructural control	How each particular design (and the attributed choreographies) partakes in forming and changing the infrastructures that enforce control on human activity. This control is manifested within the assemblage of technologies, contexts and milieus (and the related choreographic routines), cultural norms and societal powers. Infrastructural control closely relates to cultural sustainability: every design is a potential statement in regard of cultural change	– Systemic perspective that goes beyond particular situational contexts – Emphasis on cultural, societal and ethical viewpoints

In our analysis, choreography has functioned as an assistive concept. Focusing on human–technology choreographies helps to bring forth and describe the contextual effects that different devices and technological ensembles impose on our bodily doing and being. Choreographies provide a concrete way to handle embodied temporality in HCI, and they can also be flexibly applied to both objective and subjective perspectives of investigation. Thus, in addition to analyzing choreographies in objective space (e.g. as trajectories and patterns of movement), the same body movement can be treated as lived bodily experience, by conceiving of spaces in terms of the embodied engagement they constitute in a living person. For example, conceiving of the intended usage of gestural input in terms of choreographing a user's movements makes the two-directional nature of the instrumental control explicit to the designer. On the other hand, through subjectively viewed choreographies, the related design considerations of push and pull effects on the feeling of control should be more intrinsic. In regard to human–technology studies and interaction design, choreography most prominently refers to acknowledging how design choices (and the included pre-choreographies) have an effect on movements and actions while also acknowledging the existing choreographies (such as routines and social habits) of the given situations.

We propose that the choreographic approach can be useful for thinking human–technology interaction ‘out of the box’, for embracing movement especially in subjective terms of agency in lived space. In addition to focusing on devices and their use, we suggest that every design case should utilize an alternative—or parallel—standpoint with the focus on movement and the activities that people do. From the choreographic standpoint, the design of technological devices is accounted for in terms of how they partake in what we do or interface with our routine activities in the lived space. In other words, devices may be assessed in terms of how they make us move and make us feel involved, how they support our activities and even produce novel yet natural ways to engage us with the physical and social environment. As a movement-centered approach, it does not make a difference between artifacts of a different technological nature; technological devices, furniture and walls of a room can be conceptualized through choreography of the same context.

Methodologically speaking, a choreographic approach can utilize many of the current design methods in the proposed direction. This does not mean that those phenomena we intend to grasp with the presented approach would be automatically covered in the present methods—such as in applications of use scenarios (e.g. Carroll, 2000). What it does mean is that, as low-level tools, methods are applicable to the choreographic approach. It also means that through a mixed-method approach or even within a single method, the shift of focus can be strategically applied back and forth between use-oriented and choreography-oriented standpoints. For instance, use scenarios intrinsically are centered on application use, but certain segments of scenario-based design and analysis can be organized in a manner that explicitly shifts the focus to choreographies, while removing the technology to-be-designed from its focal point. Also, data from a user observation study (involving, e.g. video recordings or motion tracking) can be reflected upon the choreographic criteria. The usage of both standpoints is even recommended: although the deliberate shift from usually such a ‘magnetic’ use-orientation would reveal important phenomena and relevant layers of experience that might have otherwise remained hidden, neither of the standpoints is sufficient alone but the process of design and research needs both of them.

The essence in utilizing the choreographic approach for designing technologies of embodied control chimes with the appeal of Svanæs that a user should be seen ‘as an intelligent living body’ (2013, p. 25). Therefore, methods for inspecting choreographies need to put emphasis on rich bodily experimenting, in regard not only to the form and function of gestures, but also to the lived intentional basis and other experiential qualities in movement and actions. We see that bodystorming (Oulasvirta et al., 2003), embodied sketching (Marquez Segura et al., 2016) and any kinds of methods relying on our ‘capacity to think in movement’ (Sheets-Johnstone, 2013, p. 19) are particularly appropriate for coherently handling and making explicit both the instrumental design of bodily motions and their ‘lived aspects’. Such practices thus can simultaneously produce both quantitative (objective) and qualitative (subjective/felt) information on movements. Describing the felt dimension of movements is challenging, but detailed information on the structures and contents of particular lived experiences (such as single movements) can be obtained retrospectively by using specific explication interview techniques (Petitmengin, 2006). One should consider using sketching and the explication interview together as parallel processes (guided by an interviewer) that combine the kinesthetic creativity of embodied sketching with the elicitation of situated intentionality, feelings and mental images (relating to conceived interaction, imagined responses to the movements being sketched and experiences of control).

The significance of use scenarios lies in their potential to provide frameworks for understanding choreographies in their contexts. In terms of lived experiences, different orientational dispositions that give rise to, for instance, different push and pull-type affordances may be addressed in writing scenarios. Scenarios can also explicitly identify movement qualities and specific situational aspects (material and social) that constitute choreographies (see an example of such analysis in Loke and Kocaballi, 2016). Furthermore, we want to highlight the potential of utilizing a first-person perspective and a participatory (enactive) approach to use scenarios. A potential practical implementation of these ideas is the application of so-called rich use scenario (RUS; see Pirhonen et al., 2007). The core of RUS is to apply a first-person perspective in the form of a story, which provides a vivid imagery of the use of an application. The story can be used in participatory sessions, similar to the workshops proposed by Svanæs (2013, pp. 21–24), where movement-focused design is acted out ‘with the lived body’. The aim is to get RUS participants to immerse themselves into the story and to experience the flow of events and activities through its character. The technology to-be-designed is taken as a part of the lived environment in which the person interacts. The emphasis in the development of RUS was originally user-interface sounds, but the same approach could be applied to identifying and enacting oneself with the choreographies of a story in which actions belong to lived continuums of movement that have meaning as a larger whole. These continuums are relevant in understanding situations where contextual choreographies interface with technologies.

4. CONCLUDING REMARKS

In this paper, we presented and discussed a newly organized and coherent view of the concept of control in the context of embodied interaction and human–technology choreographies. Our intention in performing this analysis was not to produce ‘obscurantist discussion’, thus making things deliberately more complex than needed. Rather, our intention was to shed light on the phenomena of embodied interaction that too easily remain hidden or become obscured in designing technologies that tap into human movement and everyday activities. Therefore, we see that the multifaceted view of embodied control eventually provides support in design processes becoming more considered and mindful. While it provides a richer view on design possibilities, it also makes explicit the responsibilities of every designer in developing instruments of control that partake in choreographing human movements. We have suggested in this paper that the overarching utilization of the choreographic viewpoint can help designers in conceptualizing the effects that designs have on people's movements. The diversity of different design perspectives, unfolding upon the renewed definition of embodied control, highlight the need for using a heterogeneity of approaches in designing and building technologies of embodied interaction. It reminds us of Mark Weiser's 20-year-old vision of building invisible, ubiquitous technologies. A slide in his symposium keynote-presentation in 1994 states that the starting point of building these technologies is not in technologies themselves, but in ‘arts and humanities: Philosophy, Phenomenology, Anthropology, Psychology, Post-Modernism, Sociology of Science, Feminist Criticism’ and finally, ‘Your own experience…’. Below that, he added, ‘This is the most important part of the talk. You may not get it on first hearing. Patience.’ (Weiser, 1994). Today, we still need that patience.

CATEGORIES AND SUBJECT DESCRIPTORS

[Human–computer interaction (HCI)]: HCI theory, concepts and models; [Interaction Design]: Interaction design theory, concepts and paradigms.

FUNDING

This work is partly funded by the Finnish Funding Agency for Technology and Innovation (project diary number 313/31/2012) and by the Finnish Cultural Foundation.

Footnotes

1

A comparable scheme of players’ orientations with regard to musical instruments is discussed later in this paper (see Section 3.1).

2

By the way, this line of thinking seems to have similarities with the idea proposed by Godøy (2011) on conceptualizing temporal continuums of musical sound-action awareness in terms of goal-oriented, holistic chunks, into which single musical events (e.g. notes) become hierarchically subsumed.

3

This aspect seems to resonate with some of the outlined challenges (Picard, 2003) of affective computing. These challenges notably concern well-justified criticism toward handling emotions as discrete and recognizable objects or mechanical models. Relevant challenges also include the lack of understanding of situational factors of emotion expression.

4

A related conceptual framework is the one of extended cognition by Clark and Chalmers (1998; see also Clark, 2008). In it, tools—both the uses of tools as well as the results of tool use—are conceptualized as extensions of cognition. The Cartesian division into mind and body is given an extreme critique, while the borders between assumed mental events and their physical reflections are straightforwardly denied.

5

NIME also refers to an annual conference (www.nime.org) and its surrounding community of researchers and artists.

6

The term musicking is coined by ethnomusicologist Small (1998) referring to music as something we do.

Author notes

Editorial Board Member: Ian Oakley

†

Here we apologize for this small deviation from conventional grammar, but we wanted to use linguistic means to emphasize the reciprocal and ambiguous nature of control

Editorial Board Member: Ian Oakley