Virtual touch
| Adrian Cheok and Gilang Andi Pradana (2015), Scholarpedia, 10(4):32679. | doi:10.4249/scholarpedia.32679 | revision #150531 [link to/cite this article] |
Contents |
Introduction
As we move further into the digital age we are growing physically further apart from family and friends. The rapid development of society brings about a vicious cycle that can result in feelings of isolation, loneliness and a lack of sense of value ( Slater 1990 ). Research in Social Presence Theory states that less rich computer mediated communication environments inhibit communicating emotional expression, while in much richer environments in which non-verbal cues like touch are available, a full range of emotional information can be communicated due to greater social presence ( Short et al. 1976 ). While the proliferation of computers and the Internet enables us to exchange information and perform certain tasks in a quicker and more efficient manner, we are isolating ourselves from the real world where actual physical touch is very important as a communication means.
The Importance of Touch
Human touch has a long history in healing and medical therapy. In ancient Greece, Hippocrates (ca 460–370 BC), the father of Western medicine, hailed “rubbing” as an important physician’s skill ( Kline 2004 ). Modern empirical work has also shown that human touch in massage reduces stress hormones and increases the circulation of chemicals that counteract physiological arousal ( Morhenn et al. 2011 ). Simple touches, such as holding hands or tapping another’s forearm, can dispel the threat and promote calm. ( Feldman et al. 2010 ).
Touch is able to signal deeper meanings than words alone. It enables us to communicate on a social platform at deeper affectual level compared to mere words, better signaling affiliation and support. For example, while the exchange of words may vary in the greeting and farewell rituals of family members, friends and even political representatives, these rituals consistently involve tactile exchanges such as hugging, kissing or shaking hands ( Heslin et al. 1980 ). Likewise, interpersonal touch as seen in team sport accompanies or replaces verbal communication during exciting moments in the game. Touch is also important in smaller groups such as dyads, when one individual shares positive or negative news or seeks support and confirmation ( Henley 1977 ).
This problem is more pronounced for parents with young children. Children of these young ages need high care, guidance and love ( Falicov 1995 ). Parents are generally able to reach their children by telephone or video phone, but communication purely by voice or video lacks the physical interaction which has been shown in previous research to be vital in effective communication ( Bakeman et al. 1980 ). Younger children might have difficulties understanding the true meaning of words spoken by their parents. As a consequence, we require a more effective way of remote communication between parents and young children. While it may not always be possible for parents to decline work commitments (such as long office hours and business trips), remote haptic interaction may be a feasible alternative when the parent must be away from the home. Love, closeness, and intimacy are important for people’s psychological well-being ( Thoits 1985 ). If we look at another case, in couples for example, they often live apart nowadays. Accordingly, there has been a growing and flourishing interest in designing technologies that mediate a feeling of relatedness when being separated, beyond the explicit verbal communication and simple emoticons available technologies offer ( Hassenzahl et al. 2012 ).
In this chapter, we present two novel virtual touch systems for family, friends, and lovers. The first one is Huggy Pajama, a mobile and wearable human-computer-human interaction system that allows users to send and receive touch and hug interactions. Huggy Pajama consists of two physical entities. On one end, a novel hugging interface takes the form of a small, mobile doll with embedded touch and pressure sensing circuits. It is connected via the Internet to a haptic wearable pajama with embedded air pockets, heating elements and color changing fabric. Advances in mobile wearable technology have made us believe that there is a need to implement the idea into a compact wearable device. The second system is RingU, a ring-shaped wearable system aimed at promoting emotional communications in remote communication between people using the vibro-tactile and color lighting expressions. A ring is one of the fashion accessories between couples to represent their relationships. Using this metaphor, we believe that a ring is a perfect symbol of something emotionally close and connected, which fits really well with RingU aim to create a communication system that makes users feel even more connected and emotionally close. The RingU system consists of a wearable ring-shaped device and a smart phone. When a user squeezes the ring, a signal will be sent via bluetooth low energy to his/her smartphone, and then through the internet to his/her partner’s system, and it allows a virtual mini-hug and color to be sent to a paired partner’s ring. For that very instant, they will feel each other’s warm presence. The result of our experiment has shown that these augmented cues can help to stimulate a better assessment to emotional states in a computer mediated communication environment.
Touch in Human-Computer Interaction
Touch is really important in social interaction and essential in forming bonds and building trust ( Henley 1977 ), and these characteristics are also expected in virtual touch. Providing touch based interaction in Human-Computer Interaction (HCI) will increase the effectiveness in communicating emotions, building trust, and achieving behavioural changes. Humans can communicate distinct emotions through touch, and this is also possible through mediated virtual touch ( Van Erp et al. 2013 ). Touch based interaction in HCI has grown rapidly over the last few years. A range of new applications has become possible now that touch can be used as an interaction technique. One example of a system that uses tactile information is ‘inTouch’ ( Brave et al. 1997 ) first described in 1997. It introduced the method of applying haptic feedback to interpersonal communication providing a physical haptic link between users separated by distance. Another related field called Affective haptics focuses on the study and design of systems that can enhance the emotional state of a human by means of the sense of touch. Rehman and Liu from Digital Media Lab in Sweden have proposed iFeeling interface, where they implemented vibro-tactile rendering of human emotions on mobile phones, and has shown a potential to enrich mobile phones communication among the users through the touch channel ( Liu 2010 ). Another paper discovered about our readiness to empathize with and support that person by being touched by another person influences, and suggests that touch seems to be a special sensory signal that influences recipients in the absence of conscious reflection and that promotes pro social behavior ( Schirmer et al. 2011 ).
In a more closely related project, there is some work such as the CuteCircuit's Hug Shirt, which has detachable pads containing sensors which senses touch pressure, heart beat and warmth, and actuators which reproduces them. This system also utilises vibration actuators to generate the hug. Poultry Internet, ( Lee et al. 2006 ) in 2005 presented a remote human pet interaction system using a jacket specially designed with vibrotactile actuators embedded for pets. These projects indicate the attempts in the past to achieve remote haptics in close relation to hugging. Another system called HugMe ( Cha et al. 2008 ) enables touch interaction with Haptic Jacket that is synchronized with audio/visual information. The 'Hug Shirt', Poultry Internet, and HugMe use vibration to provide a sense of a remote hug which does not correspond to the feeling of natural human touch. However, through the Huggy Pajama we attempt to recreate a main property of a hug, the pressure. 'Hug over a distance' ( Mueller et al. 2005 ) uses a koala teddy to sense a hug and send it wirelessly to the air inflatable jacket to recreate a hugging feeling. The Koala teddy has a PDA in it of which the screen is touched by the user to send a hug. The PDA on the inflatable jacket upon receiving the hug activates serial controller to simulate the hug. Shifting attention to sensing technologies, we realize the importance of accurately measuring the haptic properties such as the force of a hug or touch before transmission. Even though the 'Hug over a distance' work closely relates to our concept, this system simply does not sense the force of a hug thus it simply transmits a command on an execution basis in a binary mode of "on" or "off". However, 'Huggy Pajama' is more concerned with the importance of regenerating the hugging feeling with accurate pressure levels corresponding to the input force exerted by the sender. Huggy Pajama uses the QTC sensor for accurate pressure measurement. The easy usage and manipulation of the sensor eased through the design and integration phases of the input pressure-sensing module, but the key motivation factor is the accurate pressure measurement suitable for accurate tactile sensing.
Regarding haptics in clothing, we set out to identify and compare key concepts and technologies relevant to the design space of the Huggy Pajama. Touch Sensitive by MIT Media Lab ( Vaucelle et al. 2007 ) lists down four different methods they explored for haptic apparel for massage on the move. In one prototype, thermally responsive metallic wires embedded in the apparel caused it to shrink mechanically when a current is passed through. In other prototypes, silicon buttons, vinyl inflatable air pockets and vinyl pockets filled with liquids that diffuse around a wooden ball during a massage were used. In ( Haans et al. 2007 ) the researchers used a neoprene vest with two arm straps to produce mediated touch. They also used vibrotactile actuators to enable haptic communication and have conducted a study to evaluate the effect of mediated touch. In another project ( Lindeman et al. 2004 ) the authors again use vibrotactile units to develop a haptic feedback vest to deliver haptic cues for immersive virtual environments through garments worn on the body. However many of these systems use vibrotactile actuation to enable haptics. In this project, Huggy Pajama, we stress the accurate reproduction of the pressure in a remote hugging system. As mentioned above, even though some of the systems focused on remote haptic communication, most of them focus on just the context of remote touch whereas in our project we try to recreate in high fidelity, each hug giving attention to the pressure, which is an essential property of the touch/hug. Even though most of the aforementioned works relate to remote touch, a high number of them use the vibrotactile actuation as a solution to the output haptics generation. Many of them justify this by claiming that it makes the wearable system lighter and more long lasting in terms of the battery life. However with Huggy Pajama, we employ novel techniques and equipment and use an air actuating system embedded in a jacket to exert exact amounts of pressure on the wearer simulating a realistic hug. We believe that, even though right now it may not be comparable to commercial systems in terms of the usability, this research will open up avenues for more precise communication using haptics in the future, thus enabling more effective communication of feelings.
Proposed Interface
Huggy Pajama
Huggy Pajama is a novel wearable system aimed at promoting physical interaction in remote communication between parent and child. This system enables parents and children to hug one another through a hugging interface device and a wearable, hug reproducing pajama connected through the Internet. The hug input device is a small, mobile doll with an embedded pressure sensing circuit that is able to accurately sense varying levels of pressure along the range of human touch produced from natural touch. This device sends hug signals to a haptic jacket that simulates the feeling of being hugged to the wearer. It features air pocket actuators that reproduce hug sensations, heating elements to produce warmth that accompanies hugs, and a color changing pattern and accessory to indicate distance of separation and communicate expressions.
A general overview of the system is shown in Figure 1 . On the left of the figure, an input device acts as a cute interface that allows parents to hug their child and send mood related cheerful expressions to them. On the right side of the figure, connected through the Internet, an air actuating module and color-changing clothing reproduces the hug sensation and connects the parent and child.
This pajama is able to simulate hugs to the wearer in the form of pressure that is accurately reproduced according to the inputs from the hugging interface accompanied by the generation of warmth, color changes of the fabric according to distance of separation between parent and child, as well as displaying emoticons. Our system provides a semantically meaningful interface that can be easily understood by children and parents as a reproduction of hugging. Furthermore, the hug sensation is produced in a calm and relaxing way through gentle air actuators rather than through vibration or other mechanical means. We aim to have an "impedance matching" between the input (a soft, cute, calm touch sensing interface) and output (ambient, calm, cute, hugging output).
Although never intended to replace real physical hugging, we believe this system would be of great benefit for times when the parent and child cannot be at the same physical place. Related research provides scientific evidence which showed that infant monkeys grew up to be more healthy when artificial contact comfort was given even in the total absence of their real mothers (although it would be unethical to carry out the same tests to deprive human infants artificially) ( Harlow 1958 ).
The interaction between parent and child can be bi-directional. The parent and child each can wear the pajama. Each interacts with the other through a mobile hugging interface. The bi-directionality is left as an option for the users, because at this stage, such a wearable device is not suitable to be worn at work for parents. The interactive modes are summarized in Table below.
| Interactive modes | Description |
|---|---|
| Remote touch and hug | Transmit human touch and hug on doll to wearer of haptic pajama |
| Haptic pajama | Reproduce hug sensation and warmth on wearer |
| Distance and emotion indication | Color changing clothes and accessories to give indication of separation distance between parent and child, and emotion data |
Huggy Pajama focuses on developing a mobile hug communication system for parent and child, and provides a realistic soft touch sensation. We enable users to hug or touch different areas on the hug sensing interface, and then map this to actuate different parts of the haptic pajama. Besides that, the hug sensing doll senses varying levels of force acting on it in an accurate and analog manner. The output air actuating pockets applies different levels of pressure to the human body according to the input force. Also, we experimented with color changing cloths to give an indication of distance of separation, and display emotion data of the parent and child.
In addition, our pilot study using psycho-physiological methods shows that there are no significant differences in certain aspects in comparing effects of mediated touch using our system versus real physical touch from a friend to a human participant. This is encouraging as it shows that we are able to elicit the same response from the human in remote mediated touch as compared to real physical touch.
The main functionality of the touch device is sensing the location and strength of the touch as input, and then encapsulating and transmitting this data over the internet and reproducing the force at the receiving end. Thus, it has several electronic hardware modules for each feature.
| Input touch sensing module: | is used to sense the touch levels and the area of touch of the intended recipient. The pressure variation is sensed by this module, digitized, and this information is transmitted via the Internet. |
|---|---|
| Output touch actuation module: | is used to reproduce the touch levels and positions related to the received digitized data from the input touch system. This module consists of a pneumatic system controlled electronically with air pouch actuators. |
| Fabric display module: | consists of a fabric coated with thermochromic ink and a temperature control system made from Peltier (p-n junction) cooling technology. The thermo-chromic ink color is changed according to the different levels of temperature applied. |
The overall system is a wearable remote hugging jacket which includes all three modules mentioned above. A block diagram of the mediated touch system is shown in Figure 2 . Touch sensing, touch reproduction and a color changing module with thermal control modules are connected via the Internet to reproduce real-time touch sensation and affective communication.
This system presents the flexibility for either one-way or two-way communication between the sender and the receiver. For example the sender (parent) sends a hug to their child (receiver) and if parents are at work in a business meeting, it might not be suitable for them to put on the pajama. They can easily hug their child by using only the input device. However in the case of being in office, hotel, or airport, the parent could wear the pajama and have two-way hugging with the child.
In Huggy Pajama system, there are 12 unique input sensors which corresponds to 12 unique output modules. These 12 output modules are integrated into a pajama for the children. Currently, as the result is a prototype with the need to fit all modules, we integrate them into a more sturdy soft jacket-like construction. Figure 3 shows the Huggy Pajama actual prototype in action. QTC based material was used to construct the input sensing module. We chose the QTC Sheet form factor as it allows flexibility in terms of size and shape. This allowed considerable freedom in designing the sensing component of the input interface. Figure 4 shows the touch and hug sensing and actuating device, with a zoomed in view of the individual air pouch with its controlling air actuation module.. The aesthetic design of the input device is shaped like a small doll with a body and arms along the sides to correspond to the human body where we want to reproduce the haptic sensation. The appearance of the input device is important to consider as this directly affects the user experience in affective communication and the ability to intuitively understand the system image. the system mappings and affordance. There are 12 QTC sensing areas, which cover the front and back of the doll body, as well as both the side arms. The output touch actuation system consists of air bags which inflate and deflate according to the remote input while maintaining the pressure in a constant level. Overall remote hugging jacket consists of 12 individual air pouches which corresponds to each of the 12 sensors on the input doll.
RingU
Advances in mobile technology have enabled us to effectively interact with mobile devices to do our task, mainly communication, regardless of the place. Recent technology has also made it possible to package a communication device into a range of wearable devices. We present RingU, a ring-shaped wearable system aimed at promoting emotional communications in remote communication between people using the vibro-tactile and color lighting expressions. Traditionally, a ring has been used as a symbolic present to deliver a message from the sender (the one who gives present), to the receiver. A ring is one of the fashion accessories between couples to represent their relationships. A ring is an unbroken circle, which many cultures understand as the representative of eternity, which symbolizes the eternized promise between them on their engagement and wedding. The ring can acts as a reminder and an outward symbol to others that a person is currently on an eternal commitment.
Not only for couples, ring also has a meaning as a source of unity. People wear rings to join others and symbolise that they are in the same cause. We can see some examples of the use of a purity ring, or when a group of supporters wear rings after the victory of their team. Using this metaphor, we believe that a ring is a perfect symbol of something emotionally close and connected, which fits really well with RingU aim to create a communication system that makes users feel even more connected and emotionally close.
The RingU system consists of a wearable ring-shaped device and a smart phone. When a user squeezes the ring, a signal will be sent via bluetooth low energy to his/her smartphone, and then through the internet to his/her partner’s system, and it allows a virtual mini-hug and color to be sent to a paired partner’s ring. For that very instant, they will feel each other’s warm presence, by interpreting partner's emotion from the vibrotactile and color lighting feedback. Figure 5 shows the general concept of this type of interaction in two paired rings.
Ambient multi-color glow has adopted in some research to create a certain mood and emotional feeling ( Chang et al. 2001 ). The same approach was implemented into the wearable ring, along with the vibro-tactile, which acts as a non-verbal cue in the communication channel. This system can also be integrated with social network system like Facebook, Twitter, and Google+, allowing not only one-to-one communication, but also one-to-many communication by sending a group hug to many friends in the social network at once. The scheme of RingU system is shown in Figure 6 .
The user can send 14 different combinations of lighting and 3 levels of vibration intensity through the setting of RGB color control (e.g. red, green, blue, red+green, red+blue, green+blue, red+green+blue) and vibration control ( controlling the pattern with Pulse Width Modulation ) using the smart phone app in current RingU system. Once the user sets the value of lighting and vibration expression, then he / she can send it to the partner through pressing the button from the smartphone app or squeezing the ring. By implementing force sensor on the surface of the ring, where user can do a squeezing interaction, we can measure the pressure applied by the user. The pressure exerted determines the intensity of the color and the vibration. Figure 7 shows how users can interact with the ring and control it from the smartphone app.
Several prototypes have been developed as an implementation of RingU. The prototype shown as prototype 1 in Figure 8 is the first prototype of RingU system designed for the proof of concept and it consists of a processor, a push button, an RGB LED, a vibration motor, a XBee module, and a battery. In this prototype, the push button, the LED, and the vibration motor were built into the ring and a separated box that is connected with a wire with the ring has been designed to contain comparatively bigger modules such as the processor, the Xbee, and the battery. We continued to develop RingU by implementing force sensor into the surface of the ring as an input, and making the form factor smaller, wireless, and more comfortable to wear, shown as prototype 2 and 3 inFigure 8. Currently we are developing a new design of the ring and we are rebuilding the hardware and ring design toward the aesthetic shown as prototype 4 in Figure 8.
Based on the first prototype of the RingU system, we researched how the RingU system can convey the people’s intimacy and emotional communication messages through the subtle lightings and tactile. We explored how non-verbal stimuli implemented in the ring can prime the emotion of a text message, by assigning participants to rate their emotional responses corresponding to the message, vibro-tactile stimuli, and color lighting stimuli they received during the experiment using an emotion wheel evaluation system ( Barrett et al. 1998 ). For the emotional interaction, we are continuing our study about the correlation of the emotional feedback and the various lightings and tactile expressions from the RingU system. We also are researching how tactile feedback from the ring can accompany a more meaningful transfer of emotion compared to a mobile phone.
Conclusion
We believe that computer mediated touch is an important form of human communication and will allow a major improvement in achieving meaningful remote presence. To further this goal, we have developed two remote touch systems which believe can provide great benefits in remote mediated human communication. Huggy Pajama is a novel wearable system that promotes physical interaction in remote communication between parents and children by enabling each to hug one another through a hugging interface device and a wearable, hug reproducing jacket connected through the Internet. One major contribution is the design of a remote communication system with the ability to sense and reproduce touch and hugs between two people. An additional mode of communication is provided by the incorporated cute and expressive interfaces for the conveyance of emotions between parent and children. Our second interface, RingU, is a wearable system that enables physical interaction in remote communication between loved ones. The key contribution is the ability to reproduce a mini-hug experience between two paired people remotely in a natural, physical manner. Based on this system, future works include studying and analyzing how natural mini-hugging can be generated and if it helps people’s intimate interactions while being physically separated.
References
- Bakeman, R and Brown, J V (1980). Early interaction: Consequences for social and mental development at three years. Child Development 51(2): 437-447.
- Brave, S and Dahley, A (1997). InTouch: A medium for haptic interpersonal communication. Proceedings of CHI (pp. 363-364).
- Cha, J; Eid, M; Rahal, L and Saddik, A E (2008). HugMe: An interpersonal haptic communication system. Haptic, Audio and Visual Environments and Games, 2008 (pp. 99-102).
- Chang, A et al. (2001). Lumitouch: anemotional communication device. Conference on Human Factors in Computing Systems: CHI'01 Extended Abstracts on Human Factors in Computing Systems 31: 313-314.
- Falikov, C J (1995). Training to think culturally: A multidimensional comparative framework. Family Process 34(4): 373-388.
- Feldman, R; Singer, M and Zagoory, O (2010). Touch attenuates infants’ physiological reactivity to stress. Developmental Science 13(2): 271-278.
- Feldman Barrett, L and Russel, J A (1998). Independence and bipolarity in the structure of current affect. Journal of Personality and Social Psychology 74(4): 967.
- Haans, A; de Nood, C and Ijsselsteijn, W A (2007). Investigating response similarities between real and mediated social touch: A first test. CHI '07: CHI '07 Extended Abstracts on Human Factors in Computing Systems (pp. 2405-2410).
- Harlow, H F (1958). The nature of love. First published in American Psychologist 13: 673-685. http://psychclassics.yorku.ca/Harlow/love.htm.
- Hassenzahl, M et al. (2012). All you need is love: Current strategies of mediating intimate relationships through technology. ACM Transactions on Computer-Human Interaction 19(4): 30:1-30:19.
- Henley, N (1977). Body Politics: Power, Sex and Nonverbal Communication. Prentice Hall.
- Heslin, R and Boss, D (1980). Nonverbal intimacy in airport arrival and departure. Personality and Social Psychology Bulletin 6: 248-252.
- Kline, G A (2004). The discovery, elucidation, philosophical testing and formal proof of various exceptions to medical sayings and rules. Canadian Medical Association Journal 171(12): 1491-1492.
- Lee, P et al. (2006). A mobile pet wearable computer and mixed reality system for human & poultry interaction through the internet. Personal Ubiquitous Computing 10(5): 301-317.
- Lindeman, R W; Page, R; Yanagida, Y and Sibert, J L (2004). Towards full-body haptic feedback: the design and deployment of a spatialized vibrotactile feedback system. VRST '04: Proceedings of the ACM Symposium on Virtual Reality Software and Technology (pp. 146-149).
- Liu, L (2010). iFeeling: Vibrotactile rendering of human emotions on mobile phones. The First International Workshop on Mobile Multimedia Processing, 2008, and Lecture Notes in Computer Science 5960 (pp. 1-20).
- Morhenn, V et al. (2011). Massage increases oxytocin and reduces adrenocorticotropin hormone in humans. Alternative Therapies in Health and Medicine 18(6): 11-18.
- Mueller, F et al. (2005). Hug over a distance. CHI '05: CHI '05 Extended Abstracts on Human Factors in Computing Systems (pp. 1673-1676).
- Schirmer, A et al. (2011). Squeeze me, but don't tease me: Human and mechanical touch enhance visual attention and emotion discrimination. Social Neuroscience 6(3): 219-230.
- Short, J; Williams, E and Christie, B (1976). The Social Psychology of Telecommunications. New York: Wiley.
- Slater, P (1990). The Pursuit of Loneliness. Boston: Beacon Press.
- Thoits, P A (1985). Social support and psychological well-being: Theoretical possibilities. In: I G Sarason and B R Sarason (Eds.), Social Support: Theory, Research and Applications (pp. 51-72). Boston, MA: Martinus Nijhoff.
- Van Erp, J B F and Toet, A (2013). How to touch humans: Guidelines for social agents and robots that can touch. 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 780-785).
- Vaucelle, C and Abbas, Y (2007). Touch: Sensitive apparel. CHI '07: CHI '07 Extended Abstracts on Human Factors in Computing Systems (pp. 2723-2728).
