Evaluating Information Technology Systems Using Consumer Surveys: The Role of Personal Product Knowledge

Abstract

As various types of information technology systems are becoming more pervasive than ever, many studies have evaluated the systems from the user perspective. Some of them have used surveys to measure consumers' cognitive responses to the target technology. However, this method may cause problems if the survey participants do not have a useful frame of reference for evaluating an unfamiliar system. To examine this issue, the current study empirically tested the effect of personal product knowledge on the predictability of a behavioral model, such as Technology Acceptance Model. A series of measurement invariance tests as well as multi-group comparison tests were conducted for rigorous examination of the data. Our analysis showed that the variance of attitude that is explained by the two believes (perceived usefulness and ease of use) was relatively small when the survey respondents had lower amount of product knowledge. Moreover, the group had weaker causal relationship between attitude and intention to use the technology, hindering the predictability of the research model. The results indicated that respondents should have a certain amount of knowledge of the target system in order to form accurate beliefs and behavioral decisions. The findings of this study provide important implications on sampling strategies for researchers with new technology.

1. Introduction

Over the last three decades, Technology Acceptance Model (TAM) has received extensive attention from researchers all over the world (Marangunic & Granic, 2015). As an evidence of its impact, the Google Scholar index shows that Davis’s (1989) article that introduced TAM had been cited about 40,000 times by the first half of 2018. These citations cover a broad spectrum of disciplines– not only within the neighboring fields of business administration such as management (e.g., Webster & Trevino, 1995), operations management (e.g., Boyer & Hult, 2005), accounting (e.g., Anderson & Young, 1999), human resources management (e.g., DeRosa, Hantula, Kock, & D'Arcy, 2004), and marketing (e.g., Meuter, Bitner, Ostrom, & Brown, 2005), but also beyond the traditional business disciplines in fields such as computer science (e.g., Krishnan & Kellner, 1999), communications (e.g., Trevino & Webster, 1992), medical informatics (e.g., Schectman, Schorling, Nadkarni, & Voss, 2005), education (e.g., Yong & Frank, 2003), public administration (e.g., Newcomer & Caudle, 1991), and sociology (e.g., Van den Bulte & Lilien, 2001). TAM has been also applied and validated across a vast range of system types (e.g., decision support systems, databases, expert systems, spreadsheets), cultures (e.g., East Asian vs. U.S.), and gender. Throughout its 30-year history, TAM has progressed from its initial introduction and validation through various extensions and elaborations, and in doing so has received extensive empirical support for its robustness in explaining users’ technology acceptance behaviors (Lee, Kozar, & Larson, 2003; Schepers & Wetzels, 2007).

TAM was originally designed to predict users’ acceptance behaviors of new information systems within organizations. Typically, participants in such studies are given training sessions or opportunities to learn the systems before evaluating them. Meanwhile, as information systems, including m-banking and e-learning, become more pervasive in our daily lives, TAM has been frequently applied even outside workplace settings (e.g., Arning & Ziefle, 2007; Lai & Li, 2005; Liaw & Huang, 2003). In these research contexts, some researchers seem to assume that all respondents understand the target system’s concept enough to evaluate it, but this assumption is not tenable. If survey respondents lack knowledge of the target system, this lack can cause problems with the validity of study results. This may be the reason why TAM has been criticized for its limited applicability outside the workplace (Kleijnen, de Ruyter, & Wetzels, 2007). Meuter et al. (2005) also raised a question about the validity of TAM as empirical results of testing the model have produced mixed and inconclusive results.

This study examines the effect of personal knowledge on behavioral research using TAM outside of the workplace. Specifically, we hypothesized that consumers form concrete attitudes toward a target system only when they have a certain level of knowledge of it. We tested our hypothesis using data collected from a survey about the adoption of mobile banking in Korea. The survey participants were classified into sub-groups with different degrees of system knowledge, and multi-group comparison tests with a factorial invariance test were used to compare their perceptions. The results of this study suggest that when respondents lack knowledge of the product concept, their responses must be interpreted carefully because such a condition hinders the predictability of the research model.

The paper is organized as follows. We review extant literature and present our research hypotheses in section 2. The research method is explained in section 3 and the results of our data analysis are presented in section 4. Lastly, we conclude with discussions and implications for research and practice in section 5.

2. Literature Review

To explain individuals’ technology acceptance behaviors, TAM posits that a person’s attitude toward using a target technology is influenced by that person’s perception of how easy to use and useful the technology is. This attitude toward technology adoption in turn influences the person’s intention to adopt the technology. The causal paths of TAM are shown in Figure 1.

Figure 1: Technology Acceptance Model (TAM)

TAM is an adaptation of the theory of reasoned action (TRA; Fishbein & Ajzen, 1975) that applies to the particular domain of information technology adoption and usage (Davis, Bagozzi, & Warshaw, 1989). Consequently, the theoretical causal mechanism of TRA provides the underlying theoretical explanation for the causal relationships within TAM. The basic concept of TRA is that a particular behavior is determined by an individual’s intentions to perform the said behavior. The behavioral intention is determined by the person’s attitude toward the behavior and subjective norms pertaining to the behavior in question. Attitudes toward behavior are formed by a combination of the evaluations and salient beliefs about the consequences of performing the behavior. TRA is an extremely general theory of intentional human behavior and does not specify what the set of salient beliefs is (Fishbein & Ajzen, 1975). TRA. Which salient beliefs are used in forming the attitude depends on the nature of the behavior in question.

TAM adapts TRA by specifying two salient beliefs that are consistently used in forming attitudes toward adopting and using information technologies: namely, perceived usefulness and perceived ease of use of the information technology. Specifying these two beliefs does not preclude incorporating other potentially important beliefs. Rather, the major insight and contribution of TAM is that a great portion of variance of the attitude toward adopting and using information technologies can be explained by two beliefs (Davis, 1989). This simplicity provides TAM with the parsimony that is a much desired and sought after quality for theoretical models.

Fishbein and Ajzen (1975) noted that “… a person is viewed as processing the information he has about an object in arriving at this evaluation of the object …” (p. 222). Meanwhile, the Elaboration-Likelihood Model (Petty & Cacioppo, 1981; 1986) postulates that when people have a higher ability to think about an issue, they process information more elaborately by utilizing the central route rather than the peripheral route. For example, people will process information more elaborately if they have more opportunity to think about the subject or if they are less distracted. Another important variable that influences a person’s ability to process information is the amount of prior knowledge of the target object. The degree of the ability to process information affects the extent of the information scrutiny (Petty & Cacioppo, 1986).

This idea is consistent with schema theory, which posits that people with prior knowledge are capable of using both schema-consistent and schema-inconsistent information, while people with less prior knowledge are less capable of processing schema-inconsistent information (Fiske & Taylor, 1991). Thus, if consumers lack knowledge about or familiarity with a product, they have difficulty creating an attitude toward the target product because they lack a useful frame of reference for processing the new information (Hoeffler, 2003). This difficulty is obvious when consumers evaluate a really new product (Johnson & Russo, 1984; Veryzer, 1998). In contrast, a person who has more knowledge of the target issue attempts to access more relevant internal or external sources, makes more elaborative inferences, and arrives at conclusions about the benefit of the issue (Petty & Cacioppo, 1986). With the same reason, Hoeffler (2003) and Wilton and Pessemier (1981) commented that a certain degree of knowledge is necessary for respondents to accurately assess new benefits of the product concept being studied. In the field of consumer behavior research, some researchers even oppose conducting consumer surveys to evaluate a product involving new technology due to the low predictive validity of their evaluation (e.g., Holt, 1989; Tauber, 1972).

Because the original purpose of TAM was to predict information system acceptance in workplaces, participants in surveys typically had opportunities to become familiar with the systems being studied (e.g., several weeks of use, demonstration of the system) before they evaluated it. In contrast, consumer research outside of the workplace may involve participants with little prior knowledge of the target product. The theories in psychology discussed above imply that the two salient beliefs in TAM may not effectively predict attitudes when a person has limited knowledge of a technology.

Thus, we predict that knowledge of the target information technology is a key necessary condition for people to coherently form attitudes based on perceived usefulness and perceived ease of use. Specifically, we hypothesized the following empirical results of incorporating the level of knowledge of the target information technology into TAM: the causal relationship from perceived usefulness and/or perceived ease of use to attitude would be weaker for people with less amount of knowledge of the target technology compared to those with greater amount of knowledge. If the hypothesis is supported, the two beliefs in TAM would not be able to explain enough amount of variance in attitude toward the technology for the people with less amount of product knowledge, leading to a bad model fit.

3. Research Methods and Materials

3.1. Data Collection

In order to test the impact of knowledge on the applicability and robustness of the Technology Acceptance Model, we conducted an empirical survey in South Korea. The panel database of a market research firm was used to draw a nationwide sample of adult consumers who were at least 20 years old. The age constraint was imposed to include only actual target users of banking services, both brick and mortar and/or online. The market research firm randomly sampled 1,200 members from their panel list, and these members were invited to participate in the survey via an email message that included a link to the Web-based survey questionnaire. The market research firm randomly sampled 1,200 members from their panel list and sent an email message that included a link to the Web-based survey questionnaire. We used mobile banking SERVICE1 as the target IT-enabled service for this study because consumers would have had varying levels of knowledge about such a service at the time of data collection. To set a boundary of the concept of mobile banking, the survey questionnaire briefly defined ‘mobile banking service’. The online survey was active for seven days, during which time we were able to collect a total of 346 responses from participants who had at least one personal banking account but who had not yet used mobile banking services. This approach enabled us to test the effect of consumer knowledge by controlling for the level of consumers’ direct experience with the technology. The sample demographics are presented in Table 1.

Table 1: Sample Characteristics

A survey instrument was adapted from existing measurement items in Davis (1989) and Davis et al. (1989). The wording of the scales was adapted to fit the context of mobile banking services. The instrument was then translated into Korean by a professor and later translated back into English by another professor. Differences between the original and the translated scale items were resolved through discussion. Finally, we conducted a pilot test of the translated questionnaire with 15 participants to ensure the clarity of the questionnaire items. The final instrument was produced after refining the questions based on the comments obtained from the pilot tests.

The level of personal knowledge of mobile banking was measured using self-reported items. Participants were asked to indicate their degree of agreement with two statements about their knowledge of mobile banking services –1) “I understand the basic concept of mobile banking services” and 2) “I know what is meant by mobile banking service.” Both items were measured on a 7-point Likert scale anchored from 1 = “strongly disagree” to 7 =“strongly agree.”. The reliability of the items was very high (α= 0.945).

To achieve the purpose of this study, we extracted two sub-groups with different degrees of knowledge about the use of mobile banking services from the total sample (N=346): a high- knowledge group and a low-knowledge group. For the classification, we used a median split method. The median of the responses to the knowledge questions was 5.0. If a participant’s responses to the questions were above the median, then the participant was classified into the high-knowledge group (N_high = 129), and if the responses were below the median, then the participant was classified into the low-knowledge group (N_low = 125). The high-knowledge group and low-knowledge group made up 37.28% and 36.13% of the total sample, respectively. The sample characteristics of each group are described in Table 1.

3.2. Assessment of Measurement Scales

Following the two-step approach (Anderson & Gerbing, 1988), we conducted a confirmatory factor analysis (CFA) to assess the measurement model, followed by structural equation modeling. Several indices of fit were examined to assess fit between the measurement model and the data. In particular, in addition to the traditional x² fit test, we used the goodness of fit index (GFI), the adjusted goodness of fit index (AGFI), the non-normed fit index (NNFI), the comparative fit index (CFI), the root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR). For each index, the following criteria were used to denote a good fit between the model and the data: GFI = 0.90; AGFI = 0.80; NNFI = 0.95; CFI = 0.95; RMSEA 0.06; SRMR 0.08 (Hu & Bentler, 1999). The results of CFA indicated that the measurement model fit the data satisfactorily – x² (29) = 59.173 (p < 0.001); GFI =0.967; AGFI = 0.938; NNFI = 0.979; CFI = 0.986; RMSEA =0.054; and SRMR = 0.0267.

The results of CFA indicated that the measurement model fit the data satisfactorily: x² (29) = 59.173 (p < 0.001); CFI =0.986; GFI = 0.967; AGFI = 0.938; NNFI = 0.979; RMSEA =0.054; and SRMR = 0.0267. Each item’s pattern coefficient on its theorized construct was greater than twice its standard error, thus convergent validity was supported (Anderson & Gerbing, 1988). Discriminant validity was verified if the correlation between two constructs is different from unity (Venkatraman, 1989). Within the framework of CFA, this verification can be done by comparing a constrained model in which the correlation parameters (Iij) are set to 1 with an unconstrained model in which the correlation parameters are free to vary for each pair of constructs (Jöreskog, 1971). We conducted a x² difference test for each pair of the scales and found that each pair was empirically distinct. Lastly, the reliability of the measurement scales was examined using Į coefficient, composite reliability (CR), and average variance extracted (AVE). Considering that acceptable levels of Į coefficient, CR, and AVE are 0.7, 0.7, and 0.5 or higher, respectively (Bagozzi & Yi, 1988; Fornell & Larker, 1981), the result was all satisfactory. The properties of the measurement scales are shown in Table 2.

Table 2: Properties of Measurement Scales

Significance levels: *** p < 0.001; ** p < 0.01; * p < 0.05

Note: α = α coefficient; CR = Composite Reliability; AVE = Average Variance Extracted

PEOU = Perceived Ease of Use, PU = Perceived Usefulness, Att = Attitude, BI = Behavioral Intention

3.3. Measurement Invariance Test

Before testing the main hypotheses, we conducted measurement invariance tests to validate that the survey instruments were invariant across the two groups. If there is any difference in the measurement properties used in a given study (i.e., differences in factor structure, factor loading, or measurement error) between the two groups, parameter comparison may be meaningless because research findings may be raised from the survey instruments and not the hypothesized differences – in this case, the effect of different knowledge levels (Cheung & Rensvold, 1999; Lai & Li, 2005).

The multi-group confirmatory factor analysis (CFA) feature in LISREL was used to test 1) form invariance (i.e., equality of number of factors/constructs onto which the measurement items load), 2) factorial invariance (i.e., equality of factor loadings), and 3) measurement error invariance (i.e., equality of error terms) by increasing restrictive constraints on the measurement model (Jöreskog & Sörbom, 1996). First, we built a baseline model which exhibited form invariance with four latent constructs (i.e., PEOU, PU, Att, and BI) with each measurement item loading onto its respective construct. The model fit of the baseline model was as follows: x² (58) = 107.840 (p =0.0001); CFI = 0.969; NNFI = 0.951; RMSEA = 0.0810; SRMR = 0.065). Next, we set the factor loading matrices (i.e., Λx) to be equal across groups and estimated the model to check factorial invariance. The equivalence test was performed with a x² difference test in which the above baseline model was treated as the unconstrained model and a new model with the factor loading matrices was set to be equal across the two groups as the constrained model. If the fit of the constrained model represented by the x² value is not significantly worsened from the unconstrained baseline model, the factorial equivalence constraint is reasonable (Bollen, 1989). Our analysis showed that the deterioration of fit of the model that constrained the factor loading matrix was not significant (Δx²= 5.679, Δdf = 6, p =0.460). Since equivalence of factor loadings was established, we proceeded to test for equivalence of measurement error variances. The result showed that the x² value was not significantly worsened when measurement error variance matrices (i.e. Θx) were constrained to be equal across the two groups (Δx²= 8.192, Δdf = 10, p =0.610). The hierarchical sequence of tests strongly supported the equivalence of the measurement model across the subgroups. The results are summarized in Table 3.

Table 3: Tests for Measurement Invariance

Significance levels: *** p < 0.001; ** p < 0.01; * p < 0.05

Note: Model 1 is the combined baseline model with four latent constructs. Model 2 is the constrained model with the factor loading matrix (i.e., Λ_x) set to be equal for the both low and high groups. Model 3 is the constrained model with the measurement error matrix (i.e., Θ_x) set to be equal for the both low and high groups.

4. Results

4.1. Overall Results

Before reporting on the multi-group comparison, we first present the results of estimating the overall structural model with the whole dataset (see Figure 2 and Table 4) to demonstrate what the results would have been if the analysis was conducted without considering the impact of knowledge. We also present the results of the structural model estimation for the low and high groups to show differences in structural parameter estimates and/or model fit between the overall results and the results of the subgroups (see Table 4).

Figure 2: Standardized Path Coefficients for Total Sample

Significance levels: * p < 0.05; ** p < 0.01; *** p < 0.001.

Note: PU = Perceived Usefulness; PEOU = Perceived Ease of Use; Att = Attitude; BI = Behavioral Intention

Table 4: Structural Model Estimation Results

Significance levels: *** p < 0.001; ** p < 0.01; * p < 0.05

Note: Standardized coefficients are shown with t-statistics in parenthesis.

Several results are noteworthy. First, the theoretical research model, TAM, fits the current full dataset quite well. Apart from the x² fit test (x²₍₃₀₎ = 61.6633, p < 0.001), all other fit indices suggested adequate fit between model and data. When we investigated the model fit for the sub-groups, however, a different story emerged. The model fits very well for the high-knowledge group (i.e., all fit indices indicated good fit), but very poor for the low-knowledge group, with five of seven fit indices suggesting inadequate fit between model and data (see Table 4).

Second, the model explained a significantly lower amount of variance in attitude formation from PEOU and PU in the low-knowledge group than in the full dataset or the highknowledge group (i.e., R²_total = 50.82%, R²_high = 55.00%, whereas R²_low = 32.54%). Lastly, we observed several pairs of path parameters that were not invariant across the two groups. For example, the direct path from PU to BI was not significant for the full dataset (β = 0.14, p = 0.059) or the high-knowledge group (ȕ = -0.08, p =0.489), but was significant for the low-knowledge group² (β = 0.31, p =0.005). In the next chapter, we conducted the multi-group analysis to shed some light on why such differences may have occurred.

4.2. Comparison between High and Low Knowledge Groups

Our subsequent multi-group analyses using LISREL compared each pair of parameters between the lowknowledge group and the high-knowledge group in order to determine whether each pair of the structural path coefficients was invariant across groups (i.e., H_BΓ: B_low = B_high and Γ_low = Γ_high). This step tested whether the relationships of the technology acceptance model have the same path coefficients across groups.

As shown in Table 5, the path from PEOU to PU was not significantly different across the two groups, nor was the path from PEOU to Att. The path from Att to BI was also not significantly different across the groups. Meanwhile, the path from PU to Att to was stronger for the high-knowledge group (Δx²₍₁₎ = 7.4403, p <0.01). The path from PU to BI was significant only for the low-knowledge group (Δx²₍₁₎ = 4.5777, p <0.05).

Table 5: Multiple Group Path Comparison Test

Note: Models were compared between the unconstrained baseline model and a model with a path parameter which was constrained to be equal across the two groups.

5. Conclusion and Discussions

This study examined the effect of personal product knowledge on the predictability of the behavioral research model, with an example of TAM. Several main results are noteworthy and are summarized here.

First, we observed that the model fit and parameter estimates were different when the structural models were estimated separately for different levels of target technology knowledge. More specifically, the fit of the structural model for the low-knowledge group was quite poor. To examine the source of the difference in the two model fits, we compared the amount of variances explained by the two salient beliefs between two groups. We observed that the variance of attitude that was explained by the two beliefs was relatively small for the low-knowledge group. Compared to the high-knowledge group, the low-knowledge group seemed to have some difficulty in forming their attitude based on their beliefs about the usefulness of the technology due to a lack of relevant knowledge. Specifically, people with low-knowledge had weaker relationships between perceived usefulness and attitude toward the target technology usage than did the high-knowledge group.

Instead, perceived usefulness was found to directly influence behavioral intention only for the low-knowledge group. In fact, Davis et al. (1989) acknowledged the direct influence of perceived usefulness (PU) on behavioral intention (BI), contending that in organizational settings, promotional concerns may create a direct influence from perceived usefulness to intention above and beyond the indirect impact through positive affect. Meanwhile, given the non-organizational setting of our study, it is more natural that PU does not significantly impact BI regardless of consumer knowledge level.

In consumer research of new products which have just entered the market, it is not tenable to assume that all respondents understand a product concept enough to evaluate it. Due to a lack of knowledge or previous experience, consumers have great uncertainty when they evaluate attributes of an unfamiliar product (Hoeffler, 2003). de Bont and Schoormans (1995) also recommended that when a new product concept is evaluated, only consumers with more than a moderate level of product expertise should be included because consumers with low knowledge levels may give internally inconsistent responses.

To conclude, the current study empirically supported the idea that survey respondents should have a certain amount of knowledge of the target system in order to form accurate beliefs and behavioral decisions. The result of this study cautions both academics and practitioners about sampling strategies in research using such behavioral models as the TAM. Researchers and practitioners need to make sure that participants have the required knowledge in order to form accurate beliefs and assessments of usefulness, ease of use etc. However, as noted in the introduction, it may be quite difficult in practice to ensure that participants have an appropriate level of knowledge, especially for really new products or novel innovative technologies that are in the early stages of conception and development, since there will not be many users with a requisite level of knowledge.

However, there may be instances where a technology system (or service) concept may be too new for enough people to have knowledge of it to allow reliable and valid testing of its acceptance. But even at this early stage, system (or service) designers would benefit greatly from understanding the drivers of acceptance. To accomplish this goal, researchers may need to use manipulations such as mental simulation (e.g., asking respondents to imagine they are using the product prior to administering the questionnaire) and relational analogy (i.e., asking respondents to relate the use of the product to a similar type of product they may have used before), because such manipulations have been shown to increase the accuracy of preference measurement (Hoeffler, 2003). Validating such manipulations is beyond the scope of this paper and we leave it to future research to test whether such methods may overcome the reliability and validity problems associated with low levels of knowledge about the target technology.

Endnotes

1 Mobile banking services are online services offered by financial institutions (e.g., banks) that allow both informational services (e.g., checking one’s account balance or transaction history) and transactional services (e.g., electronic bill presentment and payment [EBPP] or funds transfer) via the use of mobile devices such as cellular phones.

2 Note that given the poor fit of the overall model as suggested by the various fit indices, it is difficult to dependably interpret the path coefficient estimates. We present the results here mainly to highlight the overall differences between groups and not to state definitive results about the impact of perceived usefulness on behavioral intention for people with lesser knowledge of the target technology.