Today, many countries work on the principle that public investment in R&D is essential for the advancement of the country. In the US government’s landmark report “Science – The Endless Frontier,” Bush (1945) made a case for investment in science and ushered in a new era in which science was viewed as vital for a nation’s progress. The report led to the creation of the National Science Foundation in 1950 and government funding for R&D increased by more than a factor of ten from the 1940s to the 1960s (Pielke, 2010). During the past decades, we observe consistent growth in R&D spending globally. As research spending increases, there is also increased pressure for science to account for its accomplishments.
Initially, the aspect of interest when measuring research impact is the impact on academia and scientific knowledge. But since the 1990s, the scope of research evaluations becomes broader as the societal benefit of research has come under consideration (Bornmann, 2013). “What one expects today is measures of the impact of science on human lives and health, on organizational capacities of firms, institutional and group behaviour, on the environment, etc.” (Godin & Dore, 2005, p. 5). When finding possible societal impact indicators, researchers have recently turned to the content of social media due to the massive popularity of these sites and the ease with which data can be collected (Thelwall, Haustein, Larivière, & Sugimoto, 2013), leading to the development of a new set of social media based metrics called altmetrics.
In general, there is a growing trend of sharing research articles on social media platforms (Costas et al., 2015) and among the popular ones, Twitter was found to be the most popular, as 87% of articles that received social media mentions were on Twitter (Robinson-García et al., 2014). This is also true at a disciplinary specific level for medical and biological sciences (Thelwall, Haustein et al., 2013) and social sciences (Htoo & Na, 2017). Twitter, therefore, presents a unique opportunity to examine a large fraction of public communication about research articles.
Over the past decade, Twitter messages are increasingly being used to examine human behaviors such as smoking habits (Myslín et al., 2013), and to measure and predict real-world phenomena such as movie box office returns (Asur & Huberman, 2010) and stock markets (Bollen, Mao, & Zeng, 2011). Studying demographics of Twitter users is a common attempt to move towards more advanced observations and predictions of user behaviour in many areas. When exploring Twitter for its role in research evaluation, understanding the profiles of users tweeting research articles is necessary. As Bornmann (2014) puts it: “[w]here the broader impact of research is concerned, it is much more important to learn who has used an actual research product and why, than to simply know how many people have in total” (p. 901).
This paper seeks to contribute in this area by studying the profiles of twitter users who tweet research papers in top psychology journals. This is the extension of Na’s (2015) prior work which explored motivations and sentiments of users tweeting research articles using content analysis approach. In this paper, profiling is used as an additional factor to understand which demographics groups are involved and if there is any difference in terms of motivations in different user types. Basically, this study aims to answer the question: What are the motivations and profiles of users tweeting research articles on Twitter?
2. LITERATURE REVIEW
In the early days of citation analysis, the idea of using citation count for research evaluation and its ambiguity made the motivations behind citations a popular research subject. There were two contrasting views regarding bibliometric assessment of research based on citation. One is based on normative theory, which
argues that evaluation of science is governed by a set of norms including the acknowledgement of intellectual debt to a piece of scholarship (Merton, 1973). Those that are associated with this theory view citation as a way of acknowledgement and see citation counts as a reflection of the worth of contribution.
On the other hand, the social constructive theory suggests a different interpretation of citation. Constructivists argue that “scientific knowledge is socially constructed through the manipulation of political and financial resources and the use of rhetorical devices” (Edge, 1977; Knorr-Cetina, 1981). According to them, selection of articles to cite is largely based on the po-sition of the article’s author within the hierarchy of a scholarly community, rather than the quality of the content itself (Gilbert, 1977). The motivation is to persuade readers of the validity of their claims through authority. Latour (1987) likened the use of citations to a game.
Since the use of citation for evaluation of research performance is only appropriate when the motivations of citation are clearly understood, a large number of empirical studies attempted to reveal authors’ motivations for citing articles. Methodology-wise, these studies used (1) context or content analysis methods analyzing the semantic content of citing papers and (2) survey or interview methods identifying authors’ motives by directly surveying or interviewing the authors themselves. In both cases, a classification or taxonomy was developed to categorize the function of citations or motivations expressed by authors.
Bornmann and Daniel (2008) reviewed studies on citation behavior published from the early 1960s to mid-2005 and summarized the most common types of citation as follows:
- affirmational (citing work agrees with or is supported or influenced by cited work) ranged from about 10 to 90 percent
- assumptive (citing prior works for historical background, assumed knowledge) ranged from about 5 to 50 percent
- conceptual (use of definitions, concepts, or theories) ranged from 1 to 50 percent
- contrastive (citing alternative works) ranged from 5 to 40 percent methodological (use of materials, equipment, tools,
- analysis methods, procedures) ranged from 5 to 45 percent
- negational (negative evaluation) ranged from 1 to 15 percent
- perfunctory (citing without additional comment) ranged from 10 to 50 percent
- persuasive (citing in ceremonial fashion) ranged from 5 to 40 percent
Those findings suggest that functions and motivations behind citations vary widely and citation is certainly not an ideal indicator of research impact. However, as van Raan (2005) explained, there is enough evidence that citation motives are “not so different or randomly given to such an extent that the phenomenon of citation would lose its role as a reliable measure of impact.” The findings also provide more support for normative interpretation of citation than for social constructivist interpretation (Bornmann & Daniel, 2008) making citation a valid indicator of research performance.
As with the case of citation, finding out motivations behind a new metric is considered necessary when we try to evaluate it for its validity as a research impact indicator. As research on altmetrics grows, studies have started to look into motivations behind altmetric measures (Na & Ye, 2017; Shema, et al., 2015; Na, 2015). Among all altmetric measures, Twitter was one of the most popular as 80% of articles that were mentioned in social media were on Twitter (Htoo & Na, 2017).
Further studies on scholarly usage of Twitter provided some insights into this phenomenon. Thelwall, Tsou, Weingart, Holmberg, and Haustein (2013) conducted a content analysis of 270 tweets linking to academic articles and found 42% of tweets simply repeating the article title, but a similar number (41%) provided a brief summary of the article contents. Another similar study by Na (2015) revealed the same pattern with about 53% of 2,016 tweets containing a brief summary and about 12% containing only titles and URLs of the articles. It was also found that about 20% of the tweets are retweets.
Notable findings from this present study are that only about 7% of tweets are self-promotional in nature, tweeted either by publishers or authors of the article, and only about 3% of tweets showed negative sentiment towards the article. These findings suggest that tweet citations are much more than some publicity efforts by authors and publishers. That leads to the question: If only 7% of tweets are from publishers and authors themselves, who are tweeting and retweeting academic articles on Twitter, and why? And which demographics of society do they represent?
If tweet citations are to be used as evidence of the article’s influence on the general public or the relevance of a piece of research to the general public, then it is necessary to find out to what extent those Twitter users are representative of the general public. In this paper, we extended the study of tweet motivations by Na (2015) by finding out the profiles of Twitter users. In particular, we examined six characteristics of users: gender, geographic distribution, academic or non-academic, and individual or organization. To the best of our knowledge, this is the first study that examines the types of users who share research articles on Twitter
against their motivations.
3. RESEARCH METHODS
3.1. Data Collection
In the previous study by Na (2015), which this paper seeks to extend, articles from the top 70 journals in psychology were first chosen according to their 2013 impact factors in the Thomson Reuters Social Science Citation Index. Due to the large amount of tweet data, further data selection was done. First, all the articles were ranked by the number of mentions on Twitter. Then, recent tweets of the top ranked articles (around 133 articles) were collected from altmetric.com. For each article, up to 20 tweets in English were collected. Identical repeated tweets were removed while retweets were included. In the final selected dataset, there were a total of 2,016 tweets for analysis.
Data from altmetric.com also includes Twitter screen names of all the 2,016 tweets. Therefore, for this study, additional data, i.e. profile details of each user, were collected using the available screen names from Twitter through API. In the final dataset, name, screen name, description, profile image, time zone, location, statuses count, followers count, and friends count were included.
3.2. User Motivations
Motivations of users were the primary focus of the previous study by Na (2015). Therefore, various types of motivations were reported in a comprehensive manner with 10 main categories and 33 subcategories. Our intention in this current study, however, is to identify key motivations and their relationship with different user types. Therefore, we revised previous categories into 4 main categories (Discussion, Sharing, Promotion, and Access) and 15 sub-categories, as shown in Table 8. We maintained the coding of tweets done in the previous study by Na (2015) while we renamed and revised categories in this study. As in the original study, tweets were not mutually exclusive and a tweet can appear in more than one category or subcategory. In the table, count column shows the number of tweets in the specified motivation category or subcategory for the specified user type. Percentages in percentage column were calculated based on the total number of tweets in the respective user type as shown in Tables 4 and 5. Below is a brief description of each category and their subcategories.
Discussion: Tweets that express personal preference, personal interpretation, opinion, thoughts, criticism, and doubt were categorized under Discussion. We also kept tweets that raised questions, invited further discussion, and displayed criticism in this category. Tweets in the Discussion category were mostly different types of conversation and showed a deeper level of engagement with information in the article compared to those in the ‘Sharing’ category, where the intentions of tweets were more to share the article and related resources or to spread the findings in the article without much personal interpretations. There are six subcategories in Discussion:
D1: Expressing insights or personal interpretation of the article
Sample tweet: “Pressure’s off. Study: NO connection btw family meals & kids beh/academics. Grab a slice, mama ain’t cooking 2night! http://t.co/9Lfjly1r”
D2: Expressing personal experience or thoughts in relation to the article
Sample tweet: “Godammit they published first. I started this research shortly after puberty and now I’m beaten to the punch? WTF! http://t.co/Bau09RBo”
D3: Expressing personal preference, approval, or recommending the article
Sample tweet: “This is great social science & its free! Family meals and children’s behavioral outcomes http://t.co/oBRCRyad @WBPsychology”
D4: Raising a question to think
Sample tweet: “Sacrificing sleep time for study can backfire (diary study) - a lesson for our stuents? http://t.co/OLt5Dxzy”
D5: Inviting further discussion
Sample tweet: “Arrest followed by first contact with mental health for a good minority - prodromal symptoms? http://t.co/BeGLVsxz via @Offender_Health”
D6: Criticizing or questioning full or part of the article Sample tweet: “@bimadew Report is wishywashy, but analysis of how praise impacts learning is important, e.g. effort vs attainment. http://t.co/FjTvw2XKwy”
Sharing: Tweets in this category did not convey any opinion or interpretation but instead, the main motivation seems mainly to share the article or related resources, or spread the research findings. There are four subcategories in Sharing:
S1: Highlighting the summary or findings of the article
Sample tweet: “Prevalence of mental illness among offenders (Australian) is only 11.1%; violence=/=mental illness http://t.co/21gfpQax”
S2: Simple sharing of the article with title or part of title with the link
Sample tweet: “To Study or to Sleep? http://t.co/ZhWnUugT (Also check my Blog for more on the importance of sleep for all of us.) http://t.co/H570RY95...”
S3: Sharing to specific friends, specific groups of people, or directing the article to their followers.
Sample tweet: “@PhillipAdamsABC You might be interested in this boost for secularism— protects against depression http://t.co/VNtXlr5b”
S4: Retweeting / Heard Through (HT) / MT / via
Sample tweet: “RT @MentalHealthCop: Australian study finds offending predates onset of illness for most people with schizophrenia - http://t.co/BeGLVsxz via @Offender_Health”
S5: Sharing resources, posts, talks, activities, etc. related to the article
Sample tweet: “Due to the relevance for understanding Karl Friston’s talk at #ik2014: Clark, A. (2013) - http://t.co/kxEjE04Gr8”
Promotion: Tweets under the Promotion category were those by authors and organizations or institutions tweeting their own publications.
P1: Sharing of author’s own work
Sample tweet: “Our #NYT paper is up http://t.co/Mycamqnc”
P2: Sharing of own publications by a publisher or an
Sample tweet: “Should we believe that 1 in 68 children has #autism? Maybe not. New in our journal http://t.co/G8B6BeOeGv @DSMandell”
Access: Tweets under the Access category were those inquiring about or providing access to articles.
A1: Asking for information to access papers
Sample tweet: “Alright, this high heel study...can anyone get the paper? http://t.co/cdqu8lm4”
A2: Providing information to access papers
Sample tweet: “Here is the full article: http://t.co/ydKfsU8o”
3.3. User Types
Based on the name, description, and profile image from their Twitter profile, users were categorized into four main types (academic vs. non-academic, and individual vs. organization). The nature of users in each type is shown in Table 1. Users were categorized as unknown if the information in their profile is not clear or is missing. Similar to the classification of motivations, classifications of users into various types were not mutually exclusive and a user could be classified into more than one type. For example, a user who describes himself as a psychologist as well as researcher will appear in both academic (researcher) and non-academic (a psychologist here is considered a professional) types.
In addition to four main types, we also explored gender and geographic distribution. Individual users were categorized into male or female categories based on their first name and profile picture. In Twitter profiles, self-reported location information is available. However, we found that information is less reliable and sometimes ambiguous, such as “Earth,” “Mare Incognitum,” and “Middle of the media bubble.” Therefore, geographic information was determined based on time zone information in their Twitter profile.
3.4. Distribution of Users in Each Motivation Category
In addition to analyzing user profiles and their motivations separately, we also looked into distribution of individual vs. organization and academic vs. non-academic users in each motivation category. A chi-square test of independence was performed to examine the relation between motivations in different user types.
4. RESULTS AND DISCUSSIONS
4.1. Geographic Distribution
We began by examining geographic distribution of users sharing academic articles on Twitter. We found
Table 1. User Types
that the majority of users came from regions where English speaking countries are located. As shown in Table 2, about 86% of the users are from North America and Europe. It is most likely due to the fact that tweets in English language only were included in our sample. However, the general Twitter population at global level, as reported by Kulshrestha et al. (2012), also revealed the same pattern. In their study, with 57% of the total Twitter population, the US has the highest percentage of Twitter users, followed by the UK, the second highest with 7.3%.
This points out that, if tweets for research articles are in fact mainly in the English language, Twitter as an altmetric measure has a Western bias. Since geographic information of users and followers has important implications, more comprehensive studies are needed to further investigate this area.
4.2. Distribution of User Types
Out of the total 2,016 tweets, 1,968 tweets were identifiable as tweeted by either individuals or organizations. Table 3 shows the number of tweets and the number of unique users respectively. Out of 1,968 tweets, about 77% were tweeted by individuals while 23% were by organizations. We further divided individuals and organizations into academic and non-academic types as shown in Table 4 and Table 5. In the individual category, the number of non-academic users and tweets by non-academic users are twice more than that of academic users and tweets (Table 4).
On the other hand, in the organization category, the number of tweets by academic organizations, including publishers, was higher compared to the number by non-academic organizations, with 61% and 39% respectively (Table 5). Despite the lower number of tweets by non-academic organizations, the number of non-academic organizations is higher. Overall, the number of both tweets and Twitter users of non-academic type is higher than that of the academic type (Table 6), with 35% academic and 65% non-academic users, and 43% tweets by academic and 57% tweets by non-academic users. These findings lead to the conclusion that Twitter as an altmetric source better exposes the impact of research on the general public than on academia. In the context of this paper, the general public includes professionals, scientists, and journalists as shown in Table 1.
Table 7 shows distribution of male and female Twitter users and the number of tweets they share. It was found that the number of males and the number of tweets by males are twice more than that of females and tweets by females. This finding is in line with the
Table 2. Geographic Location of Twitter Users
Table 3. Individual vs. Organization
Table 4. Individual [Academic vs. Non-academic]
Table 5. Organizations [Academic vs. Non-academic]
Table 6. Academic vs. Non-academic
Table 7. Distribution of Gender
result from a Twitter demographic study by Murthy, Gross and Pensavalle (2016). In their study of 275 million tweets from December 2011 to March 2013 from American cities, a dominance of male users was found.
4.4. Distribution of Users in Each Motivation Category
Table 8 shows the distributions of individual vs. organization and academic vs. non-academic users in each motivation category. Our results show that, out of four main motivation categories, a large majority of tweets fell under the category of Sharing. Among the subcategories of Sharing, “S1 highlighting the summary or findingsof the article” had the highest number of tweets, with around 43% to 57% of tweets in each user types. S1 is also significantly higher than the similar subcategory “S2 - simple sharing of the article with title or part of title with the link,” which had only around 13% to 21% of tweets in each user type.
This finding revealed that the majority of users did not just copy the title of the paper but instead, they did try to find out the most significant parts (summary and findings) of the article and try to communicate it with their followers within Twitter’s limit of 140 characters. It also pointed out the fact that tweeting a paper on Twitter in most cases might not just be a mindless sharing of the article, but it carried the indication of the consumption of information, and the attempt to communicate it through a channel, which is often touted as one of the fastest sources of news. In this sense, tweets seems to be a new way of communicating research findings.
After S1 and S2, the next subcategories in rank were “S4 - Retweeting / Heard Through (HT) / MT / via” with 16% to 23% of tweets in each user type. Retweets on Twitter, if retweeted multiple times, become “a community-driven phenomenon” and, if accompanied by an appropriate hashtag, become a trending topic (Nations, 2017; Doctor, 2012). This is a way of spreading discussions in our current social media age. Retweets of academic papers therefore may be considered as indications of interest and the significance of information in the paper which is worth spreading. It is important to note here that sentiment analysis was also done on the same data set by Na (2015) and found that negative and partially negative sentiments are only around 5% in all tweets. Therefore, as with the case of traditional citation, tweet citations mainly reflect the positive impact of the paper. The last two subcategories in Sharing covered only 2% to 12% of the tweets in each user type.
A significant number of tweets were found in the Discussion category as well. The most common subcategory in Discussion is “D3 – Expressing personal preference, approval, or recommending the article,” which covered around 21% to 23% of tweets by academic users in both individuals and organizations and around 16% to 19% of tweets by non-academic users in both types. A similar subcategory is “D1- Expressing insights or personal interpretation of the article” with 3% to 7% in each user type. The remaining three subcategories covered only 1% to 7% of tweets in each user type. The least popular subcategory is “D2 - Expressing personal experience or thoughts in relation to the article” with around 0.4% to 4% users in each user type. In short, the Discussion category showed that a fair number of tweets displayed various types of engagement with articles and the most common form is expressing a positive attitude towards the articles.
The Promotion category revealed that 18% of tweets were from academic organizations, out of which about 5% were from publishers. Other than that only 0% - 5% of tweets were from individual authors promoting their own publications. Gaming and self-citation are widespread concerns when it comes to social media based metrics such as tweets. Results from this study indicated that such cases were not common enough to significantly reduce the value of tweets as a measure of impact. However, more comprehensive studies are necessary to confirm this aspect. The last category of Access covered only 0% - 5% of all tweets in each user type. While sharing information about the articles is common, as shown in the Sharing category, it seems that sharing and asking for full text articles are not common motivations when tweeting articles on Twitter.
Turning to the results of chi-square tests in Table 9, we found that there were significant differences in motivations between individual and organization users in both Discussion and Sharing categories. The main areas of difference in the Discussion category seemed to be in “D2 - expressing personal experience or thoughts in relation to the article” and “D6 - criticizing or questioning full or part of the article.” Out of all individual users, 6% (2.6%+3.8% = 6.2%) were found to express personal experiences or thoughts in relation to the article compared to only about 1.5% (0.36%+1.13%=1.49%) of organization users. Individuals also criticized or questioned full or part of the article nearly twice more. In the Sharing category, it was found that, when sharing articles, a higher proportion of individuals highlighted summaries or findings, compared to organization users. On the other hand, a higher proportion of organization users used only the article title, either full or part, when tweeting articles. Based on these findings, we can conclude that individual users demonstrated a more critical use of articles than organization users do.
Chi-square results also indicated significant differences in academic and non-academic organizations in the Discussion category. Compared to academic organizations, a higher percentage of non-academic organizations were involved in all discussion types except “D3 - expression personal preference, approval ,or recommending the article.” Between academic and non-academic individual users, academic individuals were slightly more involved in “D3 - expressing personal preference, approval, or recommending the article,” “D4 - raising a question to think,” and “D5 - inviting for further discussion,” while non-academic individuals were in “D1 - expressing insights or personal interpretation of the article,” “D2 - expressing personal experience or thoughts in relation to the article,” and “D6 - criticizing or questioning full or part of the article.” In the Sharing category, a slightly higher percentage of non-academic individuals were involved in “S3 - sharing to specific friends, groups or their followers” and “S4 - Retweeting / Heard Through (HT) / MT / via.” These differences were significant at a 10% level. In summary, there are significant but small differences in motivations between academic and non-academic individual users.
This study investigated the profiles and motivations of users sharing research articles on Twitter. From the aforementioned results, we can draw several conclusions. First, based on publicly available information on Twitter user profiles, we found that a majority of users (86%) were tweeting from North America and Europe,
Table 8. Motivations vs. Four User Types
Table 9. Chi-square Test Results
suggesting that Twitter as an alternative metric has a Western bias. Among all users, 68% were male and only 32% were female users. Second, several previous altmetrics studies suggested that tweets, and altmetrics in general, do not indicate scholarly impact due to their low correlation with citation counts. This study provides further details in this aspect by revealing that most tweets (77%) were by individual users, 67% of whom were non-academic. Therefore, tweets mostly reflect impact of research articles on the general public, rather than on academia.
Next, findings from user motivations for tweeting articles showed that a majority of tweets (from 42% to 57%) from all user types highlighted the summary or findings of the article. Thus it seems tweets are a new way of communication research findings and they also indicate the relevance of research articles to the general public to a large extent. Moreover, as we examined the differences in motivation among different user types, it was found that individual users demonstrated a more critical use of articles than organizational users do. Between academic and non-academic organizations, non-academic organizations were found to be involved
in most types of discussion rather than academic ones. However, there was no significant difference at the 5% level between academic and non-academic individuals in the way they shared articles on social media.