1. Introduction
Crowdsourcing is known as one of the “disruptive trends in technology” that is now impacting industries and conventional jobs. Disruptive technology is one that displaces an established technology and shakes up the industry or a ground-breaking product that creates a completely new industry. Disruptive technology is an innovation that dramatically changes the way consumers, businesses and industries operate. These innovations also push businesses to adapt their strategy, brace them for the possibility of losing market share or get out of date with their sector. Consequently, with the impact of COVID-19, many organizations and companies have shifted to digital technologies. Nowadays, information, communication, and technology (ICT) enable people to communicate virtually, and in particular, information technology (IT) and ICT have been used to support management, production, and even design to perform a task (Panzabekova et al., 2020).
Despite integrating with new technology, the new normal transition is not enough to ease information sharing without involving humans (Lakhwani & Omkar, 2020). Hence, the task of attracting the crowd to participate in online crowdsourcing platforms has been a straightforward approach, but sustaining them on the platform is quite challenging. Consequently, many previous studies focused on crowd workers’ motivation to participate in crowdsourcing platforms (Aitamurto et al., 2017; Hossain, 2012), but only a few studies focused on continuous motivations (Cullina et al., 2014; Deng & Joshi, 2016). Furthermore, Deng and Joshi (2016) stressed that the importance of technology in the online crowdsourcing environment. Following this gap, an in-depth investigation of technology factors that facilitate motivation among B40 crowd workers to continue using online crowdsourcing platforms has been performed (Cui et al., 2015; Deng & Joshi, 2016).
In Malaysia, crowdsourcing is a new approach (Zakariah et al., 2018) undertaken by the government and managed by MDEC (Malaysian Digital Economy Corporation) as an initiative to help the B40’s. B40’s refer to the income group in Malaysia, where B40 represents the bottom 40% of income earners (with a household income of less than RM 4, 850 per month) (Department of Statistics Malaysia, 2020). This group is categorized under the low-income group, hence, the government had introduced the crowdsourcing initiative program in 2015 to tap B40 into more Digital Malaysia programs (Zakariah et al., 2018). Consequently, the government and private sector also play crucial roles in technological advancement and product growth, constantly seeking innovative and high-efficiency innovation tools (Lee & Xuan, 2019).
Hence, this paper involves an in-depth exploration of the technology factors that facilitate B40 crowd workers’ motivation to continue their participation in online crowdsourcing platforms. The following section highlights the crowd sourcing definition and relevant literature from a technological perspective that influence crowd workers to continue using online crowdsourcing platform in performing tasks.
2. Literature Review
2.1. Crowdsourcing
Crowdsourcing was first defined as a job or task usually performed by the internal employee but then outsourced to a large group of unknown people in the form of open calls (Howe, 2006). In recent years, crowdsourcing has been used to outsource tasks to an anonymous group of individuals through online platforms (Kietzmann, 2017; Prpić et al., 2015). Crowdsourcing practices are used to gain creative ideas from the crowd in solving complex and specific problems which could benefit both the profit and non-profit industries (Brabham, 2008).
Based on this understanding, Estellés-Arolas and González-Ladrón-de-Guevara (2012) tried to integrate crowdsourcing meaning and conclude that “Crowdsourcing is a type of participative online activity in which an individual, an institution, a non-profit organization, or company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via a flexible open call, the voluntary undertaking of a task. The undertaking of the task, of variable complexity and modularity, and in which the crowd should participate bringing their work, money, knowledge and/or experience, always entails mutual benefit. The user will receive the satisfaction of a given type of need, be it economic, social recognition, self-esteem, or the development of individual skills, while the crowdsourcer will obtain and utilize to their advantage that what the user has brought to the venture, whose form will depend on the type of activity undertaken” (p. 197).
Crowdsourcing is categorized into three: job provider, a crowdsourcing platform, and crowd worker, with an additional category being added to the list, namely crowd sourcing task, which has been made famous with the four pillars of crowdsourcing (Hosseini et al., 2014). The definition of each pillar is explained in Table 1.
Table 1: Pillar of Crowdsourcing
The advancement of ICT has shifted the traditional crowdsourcing approach to online web-based and social media platforms to distribute tasks to the crowd (Feldman & Bernstein, 2014). In general, there are five types of crowdsourcing form that have been actively implemented using online web-based or social media, as listed in Table 2:
Table 2: Types of Crowdsourcing
Crowdsourcing has successfully helped job providers solve their complex tasks and save cost by breaking down the task into sub-task and distribute the tasks to the crowd. To crowd workers, digital technology has turned them to more information-intensive. Accordingly, digital technology offers contentment in working style and a better lifestyle (Panzabekova et al., 2020). Hence, crowdsourcing platforms such as Amazon Mechanical Turk (Mturk), Microworkers, Upwork.com, and Freelancer.com are among the popular marketplaces to search for crowd tasks where crowd workers are paid for every completed task. Figure 1 shows the basic flow of the crowdsourcing systems.
Figure 1: The Basic Flow of Crowdsourcing System (Source: Author’s Understanding)
The online crowdsourcing platform nowadays runs as a job matching marketplace where job providers can post their tasks and wait for crowd workers to submit their proposals. The job provider will then select the best crowd workers among the list to perform the task. Upwork.com and freelancer.com are among the online crowdsourcing platforms that run as a marketplace for crowd workers to find jobs. The diverse tasks on the platform attract crowd workers with talents and skills to perform the task they are good at. This online crowdsourcing platform operates differently from the typical online crowdsourcing platform such as crowd contest, crowdfunding, or microtask. Marketplace online crowdsourcing offers tasks ranging from complex to simple tasks such as system development, logo design, data entry, and translation with stipulated time given by the job providers to the crowd workers to complete tasks.
2.2. Technology Factors in Crowdsourcing
ICT has become a stimulus for economic development, which might be seen from the widespread impact of implementing and facilitating applications as an additional support tool for producing and consuming products and services (Agustina & Pramana, 2019). Moreover, Deng and Joshi (2016) revealed that it is vital to involve IT infrastructure when investigating crowd workers’ motivation to continue their participation in online crowd sourcing platforms. Likewise, efficient IT management should follow a proper and adequate IT infrastructure, which is necessary to encourage high performance (Lakhwani & Omkar, 2020) among crowd workers. Technology factors such as Internet connection and hardware facilities (for example, computer, laptop, and mobile phone) (Zakariah et al., 2018) are compulsory in assisting crowd workers in working better in a crowdsourcing environment. Questions have been raised on whether the way task is designed on crowdsourcing platforms would drive crowds’ participation. Besides, the lack of sufficient security mechanisms in a crowdsourcing platform can result in scam cases (Deng & Joshi, 2016).
Unfortunately, the importance of technology in crowd sourcing was neglected (Deng & Joshi, 2016) in previous literature, where researchers highlighted more on intrinsic and extrinsic motivation. Therefore, this research aimed to explore the technology factors that can facilitate crowd workers’ motivation to continue using online crowdsourcing platforms.
2.3. Theoretical Grounding Crowdsourcing
A technology study on crowdsourcing streams remains unclear (Modaresnezhad et al., 2020). Hence, because the nature of crowdsourcing revolves around field and practices closely to a natural working environment, where crowds are paid similar to the employee, this paper adopts the traditional theory by Hackman and Oldham (1976), who identified five main elements in reaching job satisfaction among workers: skill variety, autonomy, task identity, feedback, and task significance. Deng and Joshi (2016) identified digital work control (micro-task programmability, payment automation, standardization, and risk mitigation) as part of the components in understanding the crowdsourcing work environment.
In Malaysia’s perspective, the crowdsourcing term is still new (Zakariah et al., 2018), and many areas of crowdsourcing practices in Malaysia have not yet been explored, especially the opportunity to tap into the talented and skilled crowd workers to contribute to organizations or companies’ knowledge management. Hence, this paper aimed to discover the technology factors that motivate crowd workers among B40 to continue using online crowdsourcing platforms in performing tasks.
3. Research Methods
A qualitative approach was applied to explore the technological factors that facilitate crowd workers among B40s to continue using online crowdsourcing platforms. A semi-structured interview was employed where section A highlighted the demographic section and section B examined the technological factors. The qualitative research method involved steps in collecting data that were done iteratively to understand better the research phenomenon (Aspers & Corte, 2019). The iterative process involved is because of the qualitative approach’s nature which involves a non-numeric form that requires in-depth evidence and justification from the respondents (Bazeley & Jackson, 2013).
3.1. Data Collection and Analysis
A qualitative method was used in this study to explore in depth the technological factors that facilitate crowd workers among B40s to continue using online crowdsourcing platforms. Since this study focused on the low-income group in Malaysia (B40), MDEC granted permission to conduct an interview and provided respondents with the data collection process. Fourteen crowd workers (10 men and 4 women) under Global Online Workforce (GLOW) program were interviewed. GLOW is a program conducted by MDEC under national crowdsourcing initiatives to train crowd workers to earn income independently through online crowdsourcing platforms. It is an intensive training and mentoring program for qualified Malaysians to become competitive digital freelancers, winning international job and project contracts on global freelance platforms, and earning sustainable income. One of the international platforms collaborating under the GLOW program is Upwork.com. Hence, in this study, the Upwork.com platform was used as the study setting to explore the technological factors that facilitate crowd workers’ motivation to continue using online crowdsourcing platforms.
Through video conferencing, interviews were conducted using GoToMeeting software to adhere to the Movement Control Order (MCO) during the Covid-19 pandemic. Each interview session took between 45 minutes to one hour. Data collection stopped once the researchers found that the data saturation had been reached. Table 3 contains the information of the respondents:
Table 3: Participants’ Demographic
All participants were provided with an information sheet detailing the study’s purposes. The participants’ consent was collected before their inclusion in this study. Furthermore, ethical approval had been granted by the USM ethics board (JEPeM Code: USM/JEPeM/20030146) for researchers to conduct the interview. Each participant’s details were made anonymous due to their working status and individual income.
In this study, thematic analysis was used to analyze the data collected from the interview sessions. Thematic analysis is a method of analyzing qualitative data. It is usually applied to a set of texts such as interview transcripts. Researchers can identify the necessary codes and themes that could contribute to the study’s objective (Braun & Clarke, 2012). Therefore, to extract and identify the relevant codes and themes for technological factors that influence the B40 group to continue using an online crowdsourcing platform to perform the task, this study employed six phases of thematic analysis by (Braun & Clarke, 2006): (1) Become accustomed to the extracted data, (2) Identify initial codes, (3) Look for possible themes, (4) Scrutinize themes, (5) Name and describe the themes and (6) Start reporting themes. The process of analysis is compiled in Figure 2.
Figure 2: Process of Data Analysis
The interview transcriptions were analyzed and coded using Atlas.ti. The researchers iteratively cross-checked the codes and themes several times to ensure that the data extracted from the interview match the study objective.
3.2. Study Setting
The advent of paid job crowdsourcing led to creating a global online labor marketplace where resources can be purchased and delivered through platforms regardless of venue (Pongratz, 2018). Upwork then opens up a space to promote consumers in search of unique resources in their digital markets (Green et al., 2018), offering jobs such as accountants, programmers, Web developers, graphic designers, technology service professionals, and content marketers (Said, 2018). In this gig setting, companies and crowd workers may save money and reduce operating costs by accessing the resources they need through platforms. Thus, Upwork offers freelancers, entrepreneurs, and crowd workers a massive opportunity in the gig market, yet these virtues of using the website are less risky (Green et al., 2018). Hence, our study used Upwork.com as the study setting since GLOW prog has been actively using Upwork.com as their collaborative partner to train B40 crowd workers to generate income using an online crowdsourcing platform. Furthermore, all participants of this study were identified as active crowd workers on Upwork.com.
4. Findings
We identified four main themes and seven sub-themes among the three main themes for technological factors that can facilitate B40’s motivation to continue using an online crowdsourcing platform to perform the task: (1) Technology Efficacy; (2) Platform Management: Client worker management, Safety net, and Payment mechanism; (3) Platform Design: UI design, Rating feature, and (4) Infrastructure: Internet connection and Technology infrastructure. Figure 3 shows the summary of the derived themes from this study.
Figure 3: Summary of the Main Themes for Technology Factors in Crowdsourcing
4.1. Technology Efficacy
The participants in this research have pointed out technology efficacy. McDonald and Siegall (1992) defined technology as “the belief in one’s ability to successfully perform a technologically sophisticated new task” Our analysis modified the technology efficacy to match the online crowdsourcing environment as the Technology Self- Efficacy (TSE) did not underline any specific technical feature (Bandura, 1998). The participants pointed out that it is necessary to begin moving into emerging technologies to secure participation on the online crowdsourcing platform. The new technologies could be new applications where participants highlighted, updating skills in handling software is vital for them. More research and development in new technologies were necessary.
“Another thing is a skill. What we have to master is a skill in using the software. So, if I am talking about technology, it would be software skills. And for the skill, we have to make sure that our skill is updated when we work on the online platform so that we are not left behind”. (P2)
4.2. Platform Management
4.2.1. Client-worker Management
Client-worker management is a platform management sub-theme, which is a critical factor for the online system where the platform functions as a tool for the client to interact with crowd workers. Furthermore, the client-worker management factor works as a protection for both clients and workers should any issue arise. The participants also mentioned that working through such a platform trained them to deal with the job providers and enhanced the flow of communications between crowd workers and the job providers.
“And then, the platform provides two-way communications feature between job provider and the worker. We can directly deal with our job provider”. (P12)
4.2.2. Safety Net
Another sub-theme of platform management is the safety net. It is necessary to conceal (from the job providers) the personal details of crowd workers while working on the online crowdsourcing platform. Consequently, an excellent safety feature on an online crowdsourcing platform is required to secure the personal information of crowd workers. On top of that, a successful platform, according to participants, can help protects crowd workers from scammers. Hence, this finding was aligned with Lindqvist et al. (2011) who stressed in their study that when developing an online platform, security risks and matters of self-presentation should be taken into account by platform programmers.
“The platform is very strict. They have a verification function. Let’s say they have around a thousand jobs on the platform; if we click on the filter payment verified, it will display the same amount, which means the platform encourages clients to make verified payments. So, it is secure. In terms of verification, the platform is very strict”. (P4)
4.2.3. Payment Mechanism
Another sub-theme of platform management is the payment mechanism. The payment mechanism is the method by which money is transferred to crowd workers’ bank account. According to participants, working on an online crowdsourcing platform was much easier for participants to manage the money transferred from job providers to the crowd workers. Apart from the easy process of withdrawing the money from the platform to their bank account, the platform’s payment system was stable. For instance, if something went wrong, crowd workers may report the issues to the platform.
“The payment mechanism is secured, and the platform gives us the option to use PayPal, or we can choose to directly transfer the money from the platform to our bank account. So, the options available make it easier for us”. (P12)
4.3. Platform Design
4.3.1. UI Design
User interface (UI) design, the sub-theme of platform design, is one of the crucial technological factors to ensure crowd workers’ continuous participation. The participants indicated firmly that the design of features on the platform led to their desire to continue using the online platform. A previous study performed by Rahmanian and Davis (2014) stated that the success of crowd workers was affected by how the interface of the crowdsourcing platform was designed.
“The platform is very neat, well-organized, and easy to find a job. The platform is very clear. When there’s a new job posted on the platform, it will appear on the main page. And then the platform will help filter the job we want”. (P11)
4.3.2. Rating Feature
Positive feedback based on scores rated by job providers might help crowd workers feel more empowered and confident with their skills (Katmada et al., 2016). Hence, this study found that the rating feature was another important sub theme under platform design. User scores and ranking should be based upon several different dimensions of contributions to support diversification and varying skills of participants. For instance, short and long-term reputations rewarded to the existing crowd workers might promote newcomers and stimulate them while growing the crowd workers’ loyalty towards the platform. In software design, the UI was used in most studies to evaluate a system or website. However, very few studies were conducted in the context of crowdsourcing. Hence, this study identified that this feature would contribute to the crowd workers’ motivation to continue using the platform to perform the task.
“There are rating feature on the platform for job providers and crowd workers. So, we can check for valid job providers”. (P2)
“The platform has a rating feature where we can build up our profile”. (P8)
4.4. Infrastructure
4.4.1. Internet Connection
Seltzer and Mahmoudi (2013) noted several crucial issues in online crowdsourcing, such as unequal opportunities to the online world. The reason for that was because a high bandwidth was crucial to ensure crowd workers’ continuous involvement. With the current development in technologies, the Internet connection has been an important infrastructure to assist crowd workers, the job providers, and even vital to the platform management to ensure continuity in online crowdsourcing platform participation.
“We need internet, and we need a computer. Those are the most important technological tools for us. And the smartphone too. Because when we do work through online, we must have Internet”. (P12)
4.4.2. Technological Infrastructure
In addition to the Internet connection, technological infrastructure including laptops, computers, smartphones, and software might assist crowd workers when working on an online crowdsourcing platform. Working digitally, crowd workers need an appropriate gadget and the correct tools to perform the job, especially for crowd workers with graphic or software design backgrounds. The participants also stressed that they always had to ensure that their gadgets were updated when working on the platform. This factor was aligned with the finding from Deng and Joshi (2016) who concluded that technological infrastructure was an important component needed by the crowd workers to accomplish their job on the platform.
“Computer is another thing. It must be the upgraded version because it will slow down our jobs if the computer is slow. I use google docs mostly because it is much easier to submit the task on the platform later. Some workers use software tools to assist them in writing. But the most important technologies we should have are laptop and Internet”. (P14)
5. Discussion
Previous studies focused on intrinsic and extrinsic motivation yet neglecting the technological part to facilitate motivation factors. The findings of this study were aligned with the study by Deng and Joshi (2016). They highlighted the importance of technology infrastructure to assist crowdsourcing practices and ensure continuity in crowd workers’ participation on an online crowdsourcing platform. Besides that, very few studies investigated factors that influence online crowdsourcing platform’s continuous participation. In this study, themes and sub themes of technological factors to facilitate crowd workers’ motivation were merged: (1) Technology Efficacy, (2) Platform Management: Client-worker management, safety net, payment mechanism, (3) Platform Design: UI design, rating feature and (4) Infrastructure: Internet connection, technology infrastructure. Figure 4 shows the proposed guideline to understanding the technological factors that can facilitate crowd workers’ motivation to continue using online crowdsourcing platforms.
Figure 4: Summary of the Main Themes for Technology Factors in Crowdsourcing management among organizations and businesses that use
Our study concluded that technological factors should be explored in-depth to ensure the smoothness of the online crowdsourcing platform management. Additionally, systematic management might indirectly increase job performance among crowd workers and enhance knowledge online crowdsourcing platforms to tap new talent and skilled crowd workers. Furthermore, in the crowdsourcing system, the job provider must pay more attention to designing the task they wish to submit to the platform. Despite that, it was reported that the majority of the tasks posted by job providers were incorrectly designed. Hence, the finding from the previous study was aligned with the finding of this study, which highlighted the importance of platform design to ensure continuous participation among crowd workers on the online crowdsourcing platform.
Consequently, failure to design clear tasks’ objectives might confuse the crowd workers. Perhaps, a well-designed task on the platform could encourage crowd workers to stay motivated to continue using the online crowdsourcing platform. Additionally, this study’s technological factors that could facilitate crowd workers’ motivation to continue using online crowdsourcing platforms were somehow related to job satisfaction among the crowd workers. The technological factors might assist crowd workers in delivering a high-quality completed task to the client and promotes satisfaction to the crowd workers once they completed the given tasks.
From Malaysia’s perspective, this study might contribute to better crowdsourcing platform development in the future and sustain more crowd workers. Since Malaysia’s crowdsourcing initiatives were implemented to help B40 generate side income, crowdsourcing platform developers should expect non-highly technical users among the crowd workers. Hence, a well-organized and straightforward crowdsourcing platform design is very much needed in the future. Besides that, online crowdsourcing practices involving humans in conducting tasks provided by the job provider could raise issues in crowd workers’ privacy, and security since their personal information on platforms might be accessed by scammers. Hence, it is crucial to have a platform with better protection features to protect crowd workers’ personal information.
We believe that our findings in this study may contribute towards a better understanding of crowdsourcing management since it focused on crowd workers’ perspectives, not only from the job providers’ and organizations’ lenses. With the technological factors’ guideline for continuous participation on the online crowdsourcing platform proposed in this study, businesses and organizations may develop well-prepared tasks to be posted on the platform. Hence, the actions taken could result in hiring more talented crowd workers and receiving high-quality completed tasks. Furthermore, with the current pandemic COVID-19 issue worldwide, we have seen a great acknowledgment of working-from-home applied by some companies, industries, and organizations and the importance of IT infrastructure in crowdsourcing studies (Deng & Joshi, 2016).
6. Conclusion
Similar to other existing research, this study also has several limitations. First, this study was conducted from the perspective of low household income (B40) in Malaysia and does not represent crowd workers as a whole. Second, our findings were derived from the crowd workers’ experiences in using Upwork.com only. Hence, the findings of this study cannot be generalized. Moreover, future research should explore and compare the importance of each of the listed technology factors that might influence crowd workers’ motivation to continue participating in different online crowdsourcing platforms. Last, since our study was in its early stage, future researchers should pursue a follow-up study into the relationship between technical factors and motivation. Future research findings should be seen as a global standard for creating an online marketplace platform that would encourage crowd workers to continue participating in the online crowdsourcing platform.
*Acknowledgements
The authors would like to thank the Malaysia Digital Economy Corporation (MDEC) for giving information and providing participants for this research.
References
- Agustina, N., & Pramana, S. (2019). The impact of development and government expenditure for information and communication technology on Indonesian economic growth. Journal of Business, Economics and Environmental Studies, 9(4), 5-17. https://doi.org/10.13106/jbees.2019.vol9.no4.5
- Aitamurto, T., Landemore, H., & Saldivar Galli, J. (2017). Unmasking the crowd: Participants' motivation factors, expectations, and profile in crowdsourced law reform. Information Communication and Society, 20(8), 1239-1260. https://doi.org/10.1080/1369118X.2016.1228993
- Aspers, P., & Corte, U. (2019). What is qualitative in qualitative research. Qualitative Sociology, 42(2), 139-160. https://doi.org/10.1007/s11133-019-9413-7
- Bandura, A. (1998). Self-efficacy. In V. S. Ramachandran (Ed.), Encyclopedia of human behavior (Vol. 4, pp. 71-81). New York: Academic Press. https://doi.org/10.1002/9780470479216.corpsy0836
- Bazeley, P., & Jackson, K. (2013). Qualitative data analysis with NVivo. Thousand Oaks, CA: Sage Publications Limited.
- Brabham, D. C. (2008). Crowdsourcing as a model for problem-solving: An introduction and cases. Convergence: The International Journal of Research into New Media Technologies, 14(1), 75-90. https://doi.org/10.1177/1354856507084420
- Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
- Braun, V., & Clarke, V. (2012). Research designs: Quantitative, qualitative, neuropsychological, and biological. In: APA handbook of research methods in psychology (pp. 57-71). American Psychological Association.
- Chanal, V., & Caron-Fasan, M. L. (2010). The difficulties involved in developing business models open to innovation communities: The case of a crowdsourcing platform. Management, 13(4), 318-341. https://doi.org/10.3917/mana.134.0318
- Cui, T., Ye, H., Teo, H. H., & Li, J. (2015). Information technology and open innovation: A strategic alignment perspective. Information & Management, 52, 11-23. https://doi.org/10.1016/j.im.2014.12.005
- Cullina, E., Conboy, K., & Morgan, L. (2014). Crowdsourcing and crowdfunding mechanisms for scientific research funding agencies: A preliminary entity categorization matrix (PECM). http://ipp.oii.ox.ac.uk/sites/ipp/files/documents/IPP2014_Cullina_0.pdf
- Deng, X. N., & Joshi, K. D. (2016). Why individuals participate in a micro-task crowdsourcing work environment: Revealing crowd workers perceptions. Journal of the Association for Information Systems, 17(10), 648-673. https://doi.org/10.17705/1jais.00441
- Department of Statistics Malaysia. (2020). Household income and basic amenities survey report 2019. 3-57. https://www.dosm.gov.my/v1/index.php?
- Estelles-Arolas, E., & Gonzalez-Ladron-de-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information Science, 38(2), 189-200. https://doi.org/10.1177/0165551512437638
- Feldman, M., & Bernstein, A. (2014). Cognition-based task routing: Towards highly-effective task-assignments in crowdsourcing settings. ICIS-RP, 12, 1-10. https://doi.org/10.13140/2.1.1180.6722
- Green, D. D., Walker, C., Alabulththim, A., Smith, D., & Phillips, M. (2018). Fueling the gig economy: A case study evaluation of Upwork.com. Management and Economics Research Journal, 4, 104. https://doi.org/10.18639/merj.2018.04.523634
- Hackman, J. R., & Oldham, G. R. (1976). Motivation through the design of work: the test of a theory. Organizational Behavior and Human Performance, 16(2), 250-279. https://doi.org/10.1016/0030-5073(76)90016-7
- Hossain, M. (2012). Users' motivation to participate in online crowdsourcing platforms. International Conference on Innovation, Management andw Technology Research, 23(1) 310-315. https://doi.org/10.1109/ICIMTR.2012.6236409
- Hosseini, M., Phalp, K., Taylor, J., & Ali, R. (2014). The four pillars of crowdsourcing: A reference model. International Conference on Research Challenges in Information Science, 5, 1-20. https://doi.org/10.1109/RCIS.2014.6861072
- Howe, J. (2006). The rise of crowdsourcing. Wired Magazine, 14(06), 1-5. https://doi.org/10.1086/599595
- Katmada, A., Satsiou, A., & Kompatsiaris, I. (2016). Incentive mechanisms for crowdsourcing platforms. Lecture Notes in Computer Science, 99, 3-18. https://doi.org/10.1007/978-3-319-45982-0_1
- Kietzmann, J. H. (2017). Crowdsourcing: A revised definition and introduction to new research. Business Horizons, 60(2), 151-153. https://doi.org/10.1016/j.bushor.2016.10.001
- Lakhwani, M., & Omkar, D. (2020). The Impact of technology adoption on organizational productivity. Journal of Industrial Distribution & Business, 11(4), 7-18. https://doi.org/10.13106/jidb.2020.vol11.no4.7
- Lee, J. W., & Xuan, Y. (2019). Effects of technology and innovation management and total factor productivity on the economic growth of China. Journal of Asian Finance, Economics, and Business, 6(2), 63-73. https://doi.org/10.13106/jafeb.2019.vol6.no2.63
- Lindqvist, J., Cranshaw, J., Wiese, J., Hong, J., & Zimmerman, J. (2011). I'm the mayor of my house. CHI Conference on Human Factors in Computing Systems, Vancouver BC, Canada, 7 May 2011- 12 May 2011 (pp. 2409-2418). New York, US: Association for Computing Machinery. https://doi.org/10.1145/1978942.1979295
- McDonald, T., & Siegall, M. (1992). The effects of technological self-efficacy and job focus on job performance, attitudes, and withdrawal behaviors. Journal of Psychology: Interdisciplinary and Applied, 126(5), 465-475. https://doi.org/10.1080/00223980.1992.10543380
- Modaresnezhad, M., Iyer, L., Palvia, P., & Taras, V. (2020). Information technology (IT) enabled crowdsourcing: A conceptual frame-work. Information Processing and Management, 57(2), 102135. https://doi.org/10.1016/j.ipm.2019.102135
- Panzabekova, A., Kireyeva, A. A., Satybadin, A. A., & Ssabyr, N. (2020). Distribution of ICT and analysis of the digital components of the quality of life. Journal of Distribution Science, 18, 67-77. https://doi.org/10.15722/jds.18.12.202012.67
- Pongratz, H. J. (2018). Of crowds and talents: discursive constructions of global online labor. New Technology, Work and Employment, 33(1), 58-73. https://doi.org/10.1111/ntwe.12104
- Prpic, J., Shukla, P. P., Kietzmann, J. H., & McCarthy, I. P. (2015). How to work a crowd: Developing crowd capital through crowdsourcing. Business Horizons, 58(1), 77-85. https://doi.org/10.1016/j.bushor.2014.09.005
- Rahmanian, B., & Davis, J. G. (2014). User interface design for crowdsourcing systems. Journal of Advanced Visual Interfaces (AVI), 5(14), 405-408. https://doi.org/10.1145/2598153.2602248
- Said, C. (2018). Upwork debuts on Wall Street, with a freelance marketplace worth $2 billion. https://www.sfchronicle.com/business/article/Upwork-debuts-on-Wall-Street-with-freelance-13277992.php
- Seltzer, E., & Mahmoudi, D. (2013). Citizen participation, open innovation, and crowdsourcing: challenges and opportunities for planning. Journal of Planning Literature, 28(1), 3-18. https://doi.org/10.1177/0885412212469112
- Thuan, N. H., Antunes, P., & Johnstone, D. (2016). Factors influencing the decision to crowdsource: A systematic literature review. Information Systems Frontiers, 18(1), 47-68. https://doi.org/10.1007/s10796-015-9578-x
- Wazny, K. (2017). Crowdsourcing's ten years in: A review. Journal of Global Health, 7(2), 1-13. https://doi.org/10.7189/jogh.07.020601
- Whitla, P. (2009). Crowdsourcing and its application in marketing activities. Contemporary Management Research, 5(1), 15-28. https://doi.org/10.7903/cmr.1145
- Yang, K., Zhang, K., Ren, J., & Shen, X. S. (2015). Security and privacy in mobile crowdsourcing networks: Challenges and opportunities. IEEE Communications Magazine, 53(8), 75-81. https://doi.org/10.1109/MCOM.2015.7180511
- Zakariah, Z., Janom, N., & Arshad, N. H. (2018). Low-income community as crowd worker for crowdsourcing: Issues, challenges, and future direction. International Journal of Engineering and Technology(UAE), 7(4), 65-70. https://doi.org/10.14419/ijet.v7i4.33.23486