1. Introduction
Smartphones are not only devices for communication, but have also become extensions of our digital selves. People often set their own background images and specify ring tones in ways that express their identity and personality. Mobile technologies related to the presentation of a user’s identity have developed rapidly. For instance, a ringback tone service allows users to set personalized music for callers to hear while the phone rings. With the emergence of the smartphone, people are revealing themselves to others in a much richer way using multimedia presentation [1, 2]. A multimedia push ringer (MPR) enables a caller to push an outgoing multimedia ringtone to the receiving phone, which allows them to express themselves to the receiver using audio-visual content [3, 4]. Self-presentation through mobile devices affects impression formation in a variety of unexplored ways, and offers a new way to perceive and interact with others.
Recent efforts have been made to perform socio-perceptual studies in the smartphone environment [5, 6]. In particular, impression management has received mobile (Human Computer Interaction (HCI) researchers’ attention. Trait impressions using facial appearance have been extensively studied to reveal how people make judgments and decisions about others [7, 8]. These judgments can be influenced by facial properties that convey structural or emotional information [9], methods for integrating different types of cues [10, 11], and social status [12], to name a few examples. Judgments influenced by these factors are made in terms of various dimensions including attractiveness, friendliness, intelligence, competence, etc. According to Rosenberg [13], trait adjectives describing impressions of other people are clustered into two main dimensions: social and intellectual. In contrast to impression formation based on visual facial appearance, music has long been ignored in experimental studies of social cognition. Recently, researchers have begun to pay attention to the role of music in person perception or impression formation. For instance, Rentfrow and Gosling [14, 15] investigated relations between music preferences and personality. They suggested that person perception can be influenced by an individual’s music preferences. For instance, classical music fans are perceived as politically conservative and intelligent, whereas country music fans are perceived as outgoing and conventional [16]. However, there are few studies about how music might influence facial impressions. An interaction between genre of music and facial expression seems less clear, although different genres of music may contribute to enhancing or impairing perceived facial impressions.
An initial impression of others often influences preference, which may lead to a romantic or friendly relationship. One powerful factor that affects interpersonal attraction is the similarity between two people. That is, people tend to be attracted to others who are similar to, rather than dissimilar to, themselves. Similarity attraction is not limited to physical appearance, but can be extended to personality, demographics, and cultural background. However, psychological studies are limited in that participants’ personality traits are measured using personality test questionnaires, which are completed under strictly controlled experimental conditions. Thus, socio-psychological findings are seriously limited in their applicability to more practical domains.
We investigated the impact of multimedia presentation on facial impression formation using two smartphone application tools – SmartRing [17] and SystemSens+ [18]. Our research focuses on a practical attempt to study facial impression formation in a smartphone environment. We raise two interesting issues that extend impression formation into the HCI domain. First, we explore the role of a multimedia stimulus composed of music and facial images in impression formation assessed along the social warmth and intelligence dimensions. In particular, two different genres of music—classical and trot—combined with facial images were presented through SmartRing (an MPR), and impression formation was investigated across the different experimental conditions. Second, we explore the effects of similarity attraction on a facial image that can be perceived as either introverted or extroverted. The personality of the perceived face was manipulated by varying the masculinity (and femininity) of facial composites [19]. The extroversion of each participant was estimated using a regression model based on an extroversion trait test and smartphone usage patterns collected with SystemSens+.
In the following section, we present two experiments examining impression formation and interpersonal attraction in a smartphone environment. Our experimental results are discussed in the context of both interpersonal perception research and smartphone communication.
2. Facial Impression Formation with Music for Smartphones
The objective of Experiment 1 was to investigate participants’ impression of a facial image combined with one of two music genres (trot or classical). Trot is a genre of Korean pop music and is beloved by the older Korean generation, especially the working class [20][21]. In general, trot music expresses personal emotions in love and life. In some sense, trot can be compared to American country music, which represents the rural working class. In contrast, classical music is recognized as music for educated people, and is generally perceived as difficult European music. These two genres of music were selected for this experiment because the genres can be contrasted in terms of their associations with intellect (well-educated vs. less-educated) and social class (higher vs. lower). It is assumed that these genres of music may contribute differently to the formation of individual impressions.
2.1 Experiment 1-1: Evaluation of trot and classical music fans
• Participants
Fourteen participants took part in this experiment. They were paid 3000 Korean won (about 2.5 US dollars).
• Method
The most frequently downloaded smartphone ringtones in the two music genres were obtained from a Korean music website [22]. These music files were ranked on the basis of their download frequencies. Each of the 10 highly ranked files was selected as either a classical or a trot music stimulus. A highlighted section of the music files was sampled for 10 seconds and stored in a WAV file format.
The questionnaire was composed of 8 adjective words associated with the personality traits of “kindness” and “intelligence.” These adjectives include “kind,” “warm-hearted,” “ferocious,” “friendly,” “intellectual,” “smart,” “simple,” and “intelligent.” The questionnaire had 8 questions for each musical stimulus. Accordingly, the questionnaire is composed of a total of 160 items. Each item requires a numbered response from 1 to 10, representing agreement with a corresponding adjective.
Five ballad and 5 rock and roll audio clips, used as dummy stimuli, were added to the trot and classical music stimulus set. After randomly presenting a music clip, we asked participants to evaluate an imaginary fan described as liking the music.
• Results
The results [means (M) and standard deviations (SD)] are summarized in Table 1 A Student’s t-test was used to analyze the data. Two statistical values (t and p), which indicate the mean difference between two groups and the statistical significance level, respectively, were calculated. We found no significant difference between trot and classical music in the social warmth dimension [t(13) = .097, p = .924], while a significant difference was found in the intelligence dimension [t(13) = 6.962, p < .01]. This result indicates that neither trot nor classical music influences impression formation of music fans in the social warmth dimension. However, people tended to evaluate the classical music fans as more intelligent than the trot music fans.
Table 1.Perceived warmth and intelligence after presenting a genre of music
2.2 Experiment 1-2: Evaluation of facial images with the trot or classical music ringtone
• Participants
Twenty university students participated in the experiments. All participants had normal or corrected-to-normal vision and hearing. They were paid 5000 Korean won (about 4.5 US dollars).
• Apparatus: SmartRing
The multimedia stimuli composed of music ringtones and facial images was presented on an Android smartphone via a smartphone app called SmartRing. SmartRing is made for use in mobile-based technologies and allows people express themselves in the form of user-created content with various media such as text, music, images, and videos. That is, photos, music, and video clips generated by a caller can be displayed on a receiver’s phone, and vice versa. Fig. 1 graphically illustrates the SmartRing service.
Fig 1.SmartRing service. This service lets people express themselves via user-created content (UCC) using media such as text, music, images, and videos.
• Method
Facial images were obtained from the Oriental Face Database [23] and the Postech Face Database [24]. All faces in these databases are identified as Far East Asians. The databases contain hundreds of different faces. It was necessary to select only a small portion of faces for the current research. Ten postgraduate students were involved in the facial impression classification task. They were asked to classify each face in terms of social warmth and intelligence. Based on their classification, 10 facial images with a neutral expression were chosen. The ten faces roughly corresponded to the middle of the social warmth and intelligence dimensions. The images comprised 5 male faces and 5 female faces.
The music files generated in Experiment 1 were combined with the facial images. That is, the multimedia ringtones included a face and selected music files. We constructed a multimedia stimulus set that had a total of 100 ringtones (10 images × 10 music files). A multimedia ringtone was presented for 10 seconds on an experimental smartphone.
Participants were informed of the experimental procedure. Before commencing the experiment, the participants completed a practice test using a facial image presented with music. Participants were then instructed to start the experiment when they were ready. Participants were told that a multimedia ringtone would be presented on the smartphone screen through SmartRing. Ten multimedia ringtone stimuli were randomly selected and presented to participants. Ringtones with the same facial image were not presented more than once. Participants were asked to evaluate their first impression of the face, not the music.
• Results
As noted earlier, the main goal of this research was to investigate the influence of different music genres on facial impression formation. A one-way ANOVA was conducted comparing the three different conditions: image-only, with-classical-music, and with-trot-music. The descriptive statistical results (M and SD) in terms of perceived impressions for the three different conditions are summarized in Table 2 A marginally significant difference was found in the social warmth dimension (F(2, 27) = 3.01; p = .060). For social warmth, the means in the face-with-trot-music and face-with-classical-music conditions were higher than in the image-only condition. However, we failed to find any significant differences in the intelligence dimension (F(2, 27) = 0.51; p = 0.6041).
Table 2.Comparison between image-only, with-trot-music, and with-classical-music conditions along the social warmth and intelligence dimensions
3. Similarity Attraction Effects between Perceived Personality and Participant Personality
In this section, we present two experiments that demonstrate similarity attraction effects between a participant’s extroversion and the perceived extroversion of a facial image. To investigate the correlation between users' extroversion and their call profiles such as call time per day, call frequency, etc., we first carried out a correlation analysis between the Wiggins test and smartphone usage patterns collected though SystemSens+. Based on the obtained correlation, we further investigated similarity attraction effects on facial image perception.
3.1 Experiment 2-1: Correlation between Extroversion Traits and Smartphone Usage Patterns
• Participants
A total of 90 (70 male and 20 female) university students from Sungkyunkwan University participated in our experiment. All participants owned an Android smartphone, and ranged from 20 to 35 years of age. They were paid 5000 Korean won (about 4.5 US dollars).
• Apparatus: SystemSens+
In order to obtain smartphone usage patterns, participants were asked to install SystemSens+ on their smartphone. The application collects participants’ smartphone usage patterns, including the numbers of stored telephone numbers, average call time, frequency of calls and frequency of SMS. We collected smartphone usage patterns for 15 days.
• Method
Participants were asked to complete the Wiggins personality test. A modified version of the Wiggins personality test was used to measure participants' extroversion. The personality test comprises thirteen questions wherein they are asked to rate to what extent certain adjectives such as friendly, gregarious, caring, sympathetic, etc. Each question uses a 10-point Likert scale ranging from “describes very poorly” (1) to “describes very well” (10). The correlation between the obtained personality scores and collected smartphone usage patterns was measured.
• Results
Pearson’s correlation coefficients were calculated between extroversion scores and smartphone usage patterns and are presented in Table 3 We found that extroversion was significantly correlated with average call time and frequency of calls, but not with the number of stored telephone numbers or frequency of SMS.
Table 3.*p<.05 **p<.01
To estimate a user’s level of extroversion from the smartphone usage profile, a multivariate linear regression model was constructed. The results of this regression analysis revealed that frequency of calls was the most influential factor, explaining most of the variance in extroversion (β = .331). Based on this analysis, we constructed a linear regression model to estimate a participant’s extroversion as follows:
where x1 and x2 indicate the frequency of calls and frequency of SMS, respectively.
3.2 Experiment 2-2: Similarity Attraction on a Face Image Presented through SmartRing
• Participants
Forty male university students from the group of participants in the previous experiment were selected on the basis of their extroversion scores, estimated using the regression equation. Since the proportion of male students in the university is high, female participants were not recruited in order to avoid gender bias. The highest 20 and lowest 20 participants were assigned to the extrovert and introvert groups, since those falling in the middle of the extroversion scale can be considered ambiverts. The ambivert participants were excluded from this experiment.
• Method
We used 10 face images (5 male and 5 female). In order to make a face more masculine (or more feminine), we used morphing software [25], which transforms a face by altering key points on the original face to match corresponding points on the second face. The male faces were feminized and the female faces were masculinized, resulting in a facial image data set consisting of 10 original facial images and 10 manipulated facial images. Facial masculinity has been used as a cue influencing personality perception [26]. For instance, people tend to perceive a feminized male face as more introverted. Examples of manipulated faces are presented in Fig. 2.
Fig. 2.Examples of femininized and masculinized faces.
After we randomly presented the 10 facial images without repetition, participants were asked to evaluate the images in terms of extraversion and preference. They were evaluated using a 7-point Likert scale ranging from “totally disagree” (1) to “totally agree” (7) in response to adjectives or verb phrases such as assertive, extrovert, social, shy, “want to be his (her) friend,” “want to talk with him (or her),” and “want to go out with him (or her).”
• Results
We tested how participants perceived the manipulated facial images in terms of the extroversion trait using t-tests. We found that the feminized male faces were perceived to be more introverted (M = 4.596, SD = 1.1325) than were the original male faces (M = 3.93, SD = 1.0916). The mean difference between them is statistically different from zero (t(39) = -2.6807, p < 0.01). We also found that the masculinized female faces were perceived to be more extroverted (M = 4.170, SD = 1.325) than were the original male faces (M = 3.5967, SD = 1.2016). The mean difference between them is statistically different from zero (t(39) = 2.0271, p < 0.05). The results indicate that participants perceived the feminized male to be more introverted, whereas the masculinized female was perceived to be more extroverted.
We further investigated the effects of similarity attraction for the male and female faces. A two-way ANOVA was conducted on both male and female faces. The results are presented in Fig. 3.
Fig. 3.Perceived attractiveness toward male (left) and female faces (right).
For male faces, we found similarity attraction effects between estimated extroversion and male faces (F(1, 39) = 5.749, p < .05), but no main effects of estimated extroversion (F(1, 39) = .147, p = 0.703) or male face condition (F(1, 39) = .264, p = .609). That is, an extrovert is more likely to be attracted to extroverted (original) male faces, whereas introverts are more likely to be attracted to introverted (feminized) male faces. In contrast, we found no similarity effect between estimated extroversion and female faces (F(1, 39) = .212, p = .647) and no main effect of estimated extroversion on attractiveness (F(1, 39) = .096, p = .758). However, participants tended to be attracted to the introverted (original) female faces (F(1, 39) = 3.891, p = .052), not the extroverted (masculinized) female faces.
4. Discussion
The goal of the current study was to bring the social psychological issue of impression formation into the context of mobile HCI, and to demonstrate the usefulness of two smartphone apps—SmartRing and SystemSens+—that we have developed.
First, we intended to provide general guidance for how to form an impression by investigating the influence of multimedia ringback tones on impression formation along the social warmth and intelligence dimensions. Indeed, our first experiment shows that trot and classical music contribute differently to facial impression formation [27, 28]. Both music genres enhanced the perceived warmth of a person in an image, but neither significantly influenced the perceived intelligence.
In the second experiment, we focused on a different aspect of impression formation that is associated with facial attractiveness. This is an interesting issue because attractiveness matters when we want to find a romantic partner or make a friend. Our findings indicate that similarity attraction would be a good guide for constructing male–male relationships, but not for male–female relationships. Male participants tended to prefer the feminine faces to the masculinized faces. This result may reflect the Asian stereotype of an ideal woman who is polite, beautiful, and patient.
In terms of the social psychological findings, we raise some HCI issues. Current studies of impressions shaped by technology are mainly in the context of a desktop computer environment, particularly computer-mediated communication technologies [29, 30, 31]. However, technological trends are rapidly moving away from traditional computing environments toward a ubiquitous computing environment, such as the one provided by smartphones. Since smartphones are the most important communication, strategic impression management is required to deal with complex social interactions [32]. The art of self-presentation depends on displaying signs, images, and music to communicate a desired impression. In this regard, SmartRing is a useful tool to cope with the diversity of one’s social network. Even though MPRs are useful for presenting a digital self, they do not ensure that a desired impression is effectively communicated.
Similarity attraction is a strategic impression management phenomenon for establishing social relationships with others [33]. The adage “birds of a feather flock together” is frequently used to describe the similarity attraction effect. However, we may ask how we know who has the same feathers as we do. In our experiment, we used smartphone usage patterns, which can reflect an individual’s personality, collected using SystemSens+. If a user’s personality can be predicted on the basis of their usage patterns, we can manage our personal impressions by presenting a self that is similar to the person with whom we wish to affiliate. Attraction toward another person is proportional to the degree of commonality. Perceived similarity promotes social attraction, which often leads to friendship or a romantic relationship. Therefore, our work has important implications for guiding impression management in the smartphone environment.
5. Conclusion
The digital self is not an isolated cyber actor, but is shaped by interpersonal relations with others. Social impressions are formed based on how we present ourselves, and are critical in obtaining a desired outcome from the interaction. In this regard, our research extends the study of impression formation to the smartphone environment and suggests practical implications for effective impression management. However, our study has limitations and requires follow-up work. Similarity attraction effects were not fully investigated due to the imbalance in the gender of our participants. This issue will be addressed in a future study.