Users' Understanding of Search Engine Advertisements

Abstract

In this paper, a large-scale study on users' understanding of search-based advertising is presented. It is based on (1) a survey, (2) a task-based user study, and (3) an online experiment. Data were collected from 1,000 users representative of the German online population. Findings show that users generally lack an understanding of Google's business model and the workings of search-based advertising. 42% of users self-report that they either do not know that it is possible to pay Google for preferred listings for one's company on the SERPs or do not know how to distinguish between organic results and ads. In the task-based user study, we found that only 1.3 percent of participants were able to mark all areas correctly. 9.6 percent had all their identifications correct but did not mark all results they were required to mark. For none of the screenshots given were more than 35% of users able to mark all areas correctly. In the experiment, we found that users who are not able to distinguish between the two results types choose ads around twice as often as users who can recognize the ads. The implications are that models of search engine advertising and of information seeking need to be amended, and that there is a severe need for regulating search-based advertising.

1. INTRODUCTION

Search engines like Google make vast amounts of money through search-based advertisements. These are shown in response to a query, which means that the 
target of the advertisement, the search engine user, has already expressed her intent (cf. Battelle, 2005). This makes search-based ads unique to other forms of advertisements. In the context of search engines, an ad is defined as “the commercial portion of message content for which an advertiser has or will pay when a searcher sees their content after submitting a query in a search engine or Web site search box, which will typically take a searcher to another Web page” (J. Jansen, 2011, p. 232).

To understand search-based ads, we need to consider two distinct points: (1) Search-based ads are a type of search result, as they are given back in response to a query, and (2) on the search engine results pages (SERPs), there is a distinction between organic results and ads. Search-based ads are similar to organic results in that they consist of a title, a short description, and a URL, and they are also displayed on search engine results pages. Their design is also similar to organic results – the same colors or ones very similar to those used in organic results are used for headings, descriptions, and URLs. Ads may be relevant to a query. B. J. Jansen (2007) found that for e-commerce related queries, the relevance ratings for organic results and ads are practically the same. This is another argument for regarding ads as a type of results, and as not comparable to other forms of advertising. 

Ads and organic, i.e., not-paid-for results are presented on search engine results pages in different sections, and ads are labelled as being paid for. It is commonly assumed that users understand this distinction. The objective of the study presented in this paper is to test on a representative sample of the German online population whether users are indeed able to reliably differentiate between ads and organic results on Google’s search engine results pages. This article presents results from a comprehensive study on Google’s ad labelling practices. Some results have been previously published (Lewandowski, Kerkmann, Rümmele, & Sünkler, 2017; Lewandowski, Sünkler, & Kerkmann, 2017). Here, they are synthesized and discussed in a wider context. Furthermore, we present some additional results from the survey part of the study.

The study consists of three parts:

(1) Survey: We asked users about their knowledge of advertisements in search engines and their knowledge of the search engine Google and its business model.

(2) Task-based study: Users were given tasks where they had to either mark ads or organic results on screenshots of search engine results pages.

(3) Experiment: Users were randomly divided into two groups. Each participant was asked to choose one result from a SERP. Some SERPs contained advertisements, and some did not, which allowed us to determine whether advertisements influenced users’ selections.

The contributions this study makes are the following: First, through a large-scale descriptive study, we can derive valid results on users’ understanding of search engine ads, differentiating our study from small-scale studies and studies using convenience samples. Second, we strengthen the external validity of our results through triangulating results from a survey, a taskbased study, and an online experiment. Third, our study contributes to theory in that it gives clear indication that information seeking models are incomplete, as they do not consider users who are unable to make informed decisions when it comes to results selection. Fourth, our study contributes to policy as it gives clear indication that regulating bodies need to become active in forcing search engine to label search engine ads in a way that allows users to clearly distinguish them from the organic, i.e., non-paid results.

Our research resides within the context of the influence search engines have on what information users get to see (see Goel, Broder, Gabrilovich, & Pang, 2010; Höchstötter & Lewandowski, 2009), how information literate users are in their use of search engines (see Purcell, Brenner, & Raine, 2012; Rowlands et al., 2008), and whether dominant players in the search engine market should be regulated (see Vogl & Barret, 2010), and if so, in which regard (see Lewandowski 2014)

The rest of this paper is structured as follows: First, we present a literature review on the relevant prior work in the areas dealt with in this paper. Then, we present the method of data collection for our studies. After that, results from the survey, from the task-based study, and the online experiment are presented. We discuss these results in detail and conclude by putting the result into context and giving some directions for future research.

 

2. PRIOR WORK

In this section, we report on prior work in the following areas: Design of search engine results pages and its influence on user behavior, advertising as the main 
source of revenues for search engine vendors, and ads labelling. A more general approach to understanding search engine advertising is presented in J. Jansen (2011) whose work, however, focuses on the side of the advertisers and discusses user behavior from that perspective.

 

2.1. Design of Search Engine Results Pages and Its Influence on User Behavior

Modern search engine results pages consist of different elements and results ypes. While the results presentation in search engines had been strictly list-based in the past, for the last ten years or so we have witnessed a fundamental change in results presentation: While the ranked list of organic results is still a central element to the SERP, other results (and new forms of results presentation) gain more and more importance (Tober, Grundmann, & Thakur, 2016). For an overview of different SERP designs, sizes, and the effect they have on selection behavior, see Kelly and Azzopardi (2015).Four major results types can be found on the SERPs (taken from Lewandowski, Kerkmann, et al., 2017):

• Organic results are results that are generated from the search engine’s index of web pages. Every document in that index has the same chance of being displayed in response to a specific query, as all documents are treated the same by the ranking algorithms.
• Advertisements in the context of search engines are text-based (as opposed to graphical banner ads on the internet). They are also shown as a response to a query and form a separate results list (or more than one separate results list) on the SERP. 
• Universal Search results are results generated from vertical search engine indexes, such as news, video, or images. Depending on the nature of the index, these results can either be generated similarly to organic results (as in the case of images) or be based on a specific collection of sources (as in the case of news, where a collection of trusted news sources is defined beforehand by the search engine vendor). Universal Search results can also come from document collections especially built by the search engine vendor (as opposed to the results from the web index that come from a multitude of sources distributed across the web). 
• Knowledge-graph results are results within which the search engine displays actual answers or facts instead of presenting links on the SERPs. Thus, if users find answers to their search queries within the knowledge graph results, they do not need to click through to a results document.” 

 

It is important to know that the design of search engine results pages, i.e., the way and order in which results of different types are presented, directly influences users’ viewing and section behavior: Users focus (and click) on results that appear at or near the top of the first page; users tend to click on results at or near the top of a results list (Joachims, Granka, Pan, Hembrooke, & Gay, 2005). For instance, a 2014 study from Caphyon used 465,000 queries and analyzed the click-through rates in Google. They found that more than two-thirds of all clicks go to the first five positions, and the result ranked first alone accounts for 31% of all clicks (Petrescu, 2014). This clicking behavior is because, usually, lists of results are read from top to bottom, so that users pay little or no attention to items shown lower down. Either the user thinks that results positioned higher up are more relevant or the user finds what he or she is looking for before perusing all the results. This positioning bias also arises in connection with other types of lists. This, however, does not mean that users never consider results other than those displayed at the top. Instead, it means that considering further results is often unnecessary for a user who has already found something (that at least seems to be) relevant to their query among the first few results.

Users tend to focus (and click) on results that appear above the fold. It is essential to bear in mind whether a search engine result appears above the so-called “fold,” 
i.e., in the area of the search engine results pages that is immediately visible without the need for scrolling. That is because appearing above the fold increases the probability that a user will click on a link to a Web page (Cutrell & Guan, 2007; Granka, Hembrooke, & Gay, 2006; Joachims et al., 2005). The corollary is also true, that is to say, results below the fold are not immediately visible to users and are less likely to be clicked on.

When results are presented larger (and therefore occupy more of the “screen real estate”; see Nicholson et al., 2006), the probability of them being viewed by users and clicked on gets higher (Z. Liu, Liu, Zhou, Zhang, & Ma, 2015). Users tend to look at and click on results displayed with large or attractive graphical features: When graphics or images are used for results presentation, the user’s gaze is led towards these results (Z. Liu et al., 2015). Results presented in this way are more likely to be clicked on by users. Richer representation of results in snippets on the SERPs leads to some snippets (1) occupying more screen real estate and (2) being more attractive due to thumbnail images, which, in turn, attracts more clicks by users.

Search engine users trust the search engine to produce relevant results and to put the most relevant results at the top of the first SERP. Studies have shown that when considering results lists, users trust Google’s ranking even more than their own judgments (Pan et al., 2007). This means that results shown on lower positions of the results list are regarded as less trustworthy, regardless of their actual relevance to a particular search query or trustworthiness. Tremel (2010) summarizes the results of an extensive study on user views about the credibility of search engine results: “Users apparently [follow] for the most part the machine and its relevancy assessment, which is implicitly communicated by the ranking of the hits” (p. 249, translated from German).

 

2.2. Advertising as the Main Source of Revenues for Search Engine Vendors

Search engine advertising is the central business model of all Web search engine providers. For instance, Alphabet, Inc. (Google’s parent company) had a total 
revenue of 90.27 billion U.S. dollar in 2016 (Alphabet Inc., 2017), of which 88.9 percent was generated by selling advertisements. These revenues are further divided between AdWords, i.e., ads that are shown on search engine results pages as a response to a query, and AdSense, i.e., ads that are generated based on the textual content on third-party content pages. Approximately 80 percent of Alphabet’s advertising revenue is generated from search-based ads. It is therefore obvious that this latter form of advertising builds the backbone of Google/Alphabet and that the company has a huge interest in users clicking ads. This leads to a situation where search engine vendors may be tempted to blur the line between advertisements and organic results to make more users click on ads and therefore increase their revenues. Figure 1 shows two results for the same query and the same website, one being an ad and one being an organic result. One can see that the results presentation is very 

 

Fig. 1. Result description (snippet) of an ad vs. an organic result 

 

similar and that the main difference is the labelling with the word “Anzeige” (German for “ad”).

Other search engine companies conduct similar practices as Google when it comes to advertising. In this article, we focus solely on Google because of its overwhelming market share, which is more than two thirds in the U.S. and well over 90 percent in most European countries. 

The success of contextual advertising in search engines can be explained by five factors (Lewandowski, 2015, p. 149):

(1) By submitting search requests, users reveal their interest in specific topics. This type of promotion, therefore, results in little to no wasted advertising, since the search request indicates a certain level of interest on the part of the user. Simultaneously, the context-based nature of the advertisements means that whenever little to no commercial opportunity exists, no advertisements are displayed. Consequently, searchers are not burdened with ads in such cases.

(2) Ad buyers are billed based on the number of clicks. Purchasers of this type of advertising thus only pay if a search user shows actual interest in the advertisement by selecting it on the search engine results page. This, in turn, reduces wasted advertising expenditures.

(3) Contextual advertising in search engines is textbased and thus less distracting for users than other graphical types of advertising.

(4) The auctioning procedure with which advertising customers bid against one another for each click on every search term ensures transparent prices that are based on the actual competition for each search term. 

(5) Advertising customers can easily create advertisements or advertising campaigns themselves online without needing to hire an ad agency. Due to the pay-per-click (PPC) payment model with relatively low prices for a click and the ability to set a daily limit, this type of advertising can also be used successfully by companies with only a limited advertising budget. Also, the self-service nature of the system allows unsuccessful advertisements to be quickly and easily revised. Multiple versions of an advertisement can also be tested live.

Contextual advertising thus offers advantages for both users of search and advertisers. For search engine users, contextual advertising is less distracting – due 
also to the fact that it is only displayed in the event of a presumed interest. The most significant advantage for advertisers is in the precise control of how their advertising is delivered.

Regarding the presentation of advertisements on search engine results pages, an early study by Nicholson et al. (2006) found that only 40% of the results on the first screen (i.e., the area visible to the searcher without scrolling down) were organic results. This ratio increased to 67% when considering the first results page. This study, however, is limited by the low number of queries used and the fact that only one (rather low) screen resolution was considered. However, it clearly shows the problems search engines face when the screen size is limited: To have users click on the ads, they need to be positioned first, which may lead to organic results being ranked down. This could even result in search engine results pages where, in the area above the fold, only advertisements are presented. Such an example is shown in Figure 2: In the mobile version of the SERP, a user needs to scroll down to reach the first organic result.

 

Ad labelling

Considering the four areas of the SERPs discussed above, it may be hard for users to distinguish between elements which were produced by objective criteria and 
elements which are produced by commercial decisions. When considering the different results types, especially organic results and Universal Search results, there may be blurred lines between paid and unpaid results, especially when Universal Search results point to offerings

 

Fig. 2. Google search engine results pages showing ads and organic results; desktop version vs. mobile version

 

by the search engine itself (e.g., results for flight searches where Google does not only display results but also makes it possible to directly book a flight, a transaction 
from which Google gets a commission) 

An early study (Fallows, 2005) found that only 38% of U.S. searchers were aware of the distinction between organic and paid results. The situation has undoubtedly 
changed since then, partly due to the U.S. Federal Trade Commission’s guidelines on search engine ad disclosure (Sullivan, 2013a). However, the distinction between 
the two results types is still an issue, not only in general-purpose Web search engines but also in many specialized vertical search engines (Sullivan, 2013b). Some industry studies strongly suggest that the labelling of ads might not be clear enough (Bundesverband Digitale Wirtschaft, 2009; Charlton, 2013; Wall, 2012). Furthermore, studies found that ads placement and labelling influence users’ viewing patterns and selection behavior (Z. Liu, Liu, Zhang, & Ma, 2014), and that different variants of ads labelling lead to different perceptions of whether a result is an ad or not (Edelman & Gilchrist, 2012). However, these findings either come from small-scale studies or studies using non-representative samples of search engine users. While experimental studies testing different forms of labelling can draw conclusions on the actual effects these forms have, they are restricted by the user sample used, i.e., they lack external validity, as it is unclear whether we can extrapolate from the sample to the larger population.

 

3. DATA COLLECTION

For our study, we collected our data using a representative sample of the German online population. The sample was built according to AGOF criteria (‘Methode - Reichweitenwährung der AGOF,’ 2015) and consisted of 1,000 users. AGOF provides a standardized online coverage currency to measure the success of marketing tools. The online coverage currency is based on a Three-Pillar Model for data mining and profiling by electronic measurement of page visits and page impressions, by on-site surveys on descriptive socio-demographic values, and by representative telephone surveys. The population includes Internet users from the age of 10 years on. The sampling is based on recruiting participants from different slices of the sample. They are invited until the desired number of participants in each slice is reached. Due to the AGOF procedures, we are unfortunately not able to give numbers on how many users were asked to participate but declined. A market 
research firm carried out the survey in December 2013.

 

4. SURVEY

In the survey part of the study, we asked users about their knowledge of Google’s business model. The aim was to find out whether users understand how search 
engines and Google, respectively, make their money and whether they understand advertisements as the search engines’ business model.

The questions were as follows:

1. Have you used the Google search engine within the last three months? – This was to distinguish Google users from non-users.

2. How would you rate your competencies when it comes to searching in Google? Please use school grades.

3. Search engines are commercial Internet services,and therefore need to make money. Please describe in your own words how the search engine Google generates its revenues.

4. Is it possible to pay Google for preferably listing one’s company on the search results pages, as an answer to a search query?

The following questions were only given to users who had answered question 4 with “yes”:

5. Is it possible to distinguish between paid advertisements and unpaid results on Google’s search engine results pages?

6. How do paid advertisements differ from unpaid results? Please name the most important differences.

The answers to the first question, “Have you used the Google search engine within the last three months?”, produced uniform results: All participants had used that search engine within the time-span mentioned. With the second question, we asked the participants to rate their competency when it comes to searching in Google. We used the German grading system, where 1 is the best grade and 6 the worst. The vast majority (90.8%) rated themselves as either “very good” (grade 1; 45.4%) or “good” (grade 2; 45.4% as well). There were only a few users who rated their competency with a 4 or worse. This confirms earlier findings that users feel confident when using search engines and are convinced that their skills are sufficient for effectively using them (see Purcell et al., 2012, p. 13).

Next, we asked participants to describe in their own words how Google generates its revenue. This was an open question, i.e., users filled in their own answers, which were later classified by a research assistant. 81% correctly named advertising as the source of Google’s revenues. However, only 60.6% named advertising as Google’s only source of revenue, and 20.4% mentioned other, incorrect sources of revenue. 9.5% gave an outright wrong answer, and another 9.5% said they did not know.

In question 4, we asked users whether it was possible to pay Google for giving one’s company a preferred listing on the search results pages for a particular search query. With the wording of this question, we wanted to make sure that, hypothetically, both paid placements in organic results and placements of contextual advertisements were covered. 73.3% of participants correctly said that such a listing was possible. 6.4% said this was not possible, while 20.3% said they did not know. In an earlier (small-scale) German study asking the same question, only 64% said yes, while 36% said this was not possible (Bundesverband Digitale Wirtschaft, 2009).

We then asked the users answering the last question with “yes” whether it was possible to distinguish between paid advertisements and unpaid results on Google’s search engine results pages. 57.98% said this was possible, while 26.6% said it was not, and 15.42% said they did not know.

Taken the last two questions together, we can see that 42% of the German Internet users self-report that they either do not know that it is possible to pay Google for 
preferred listings for one’s company on the SERPs or do not know how to distinguish between organic results and ads.

With question 6, we asked participants who said that it was possible to distinguish between paid advertisements and unpaid results, in which way these two results types are differentiated. This again was an open question where we asked the participants to name the most important differences. Again, a research assistant classified the answers (see Table 1). Note that it was possible to mention more than one option.

As can be seen from Table 1, there were only 37 participants who named the elements distinguishing ads from organic results (at the time of data collection) correctly, namely:

1. Shading and/or different layout
2. Shown at the top of the results page, before the organic results
3. Shown on the right-hand side of the search engine results page

Note that with this question, we asked for the general characteristics of the ads. However, on some SERPs, ads are only shown on the top of the organic results pages or on the righthand side respectively.

Many users named some of the correct elements, but not the complete set. Numbers for the different combinations can again be found in Table 1.

In Figure 3, the segmentation of users from questions 4 to 6 is summarized. In total, 62.2% of participants were not able to distinguish between ads and organic results. They either said it was not possible to pay Google for a preferred listing of one’s company on the SERPs, that they did not know if this was possible, or they said that it was not possible to distinguish between ads and organic results (or they did not know), or they named incorrect ad labelling (or did not know). This number 
does not include the 341 users who named some correct elements for identifying ads, but not the complete set of distinguishing elements.

 

5. TASK-BASED STUDY

The task-based study is based on five tasks where users had to either select all areas where they assumed there were advertisements or all areas where they assumed there were no advertisements. The stimulus material was search engine results pages with different layouts and elements. We tried to find a sample of SERPs that matched the typical results presentations at the time the study was conducted. Information on the types of results presented on each SERP used can be found in Table 2.

We further used queries of different types. We classified queries according to Broder’s taxonomy into informational, navigational, and transactional (see Broder, 2002). In the present study, however, we did not use navigational queries, as we wanted our participants to be able to choose from the whole set of results provided and not to already point them to a specific result. We further divided the queries according to the user’s intent into informational and commercial (see Lewandowski, Drechsler, & von Mach, 2012). This was explicitly written into the task descriptions, as to prime participants on whether they were interested in buying something (questions 9 and 11) or finding information on a topic (questions 8 and 10). An overview of the elements used on the SERP screenshots can be found in Table 3.

In the first task, users saw a pixelated screenshot of a search engine results page from Google, where they had to select all areas where they assumed there were advertisements. The aim of this task was to find out if users were able to distinguish between organic results and advertisements solely on the basis of the structure of the SERP.

For the remaining four tasks, we used screenshots of Google search engine results pages (see Figure 4). Again, participants had to select all areas where they assumed there were advertisements or no advertisements, respectively. The tasks were as follows:

Table 1. Answers Given by Participants for Combinations of Possible Options Displayed (Excluding Possible Combinations Not Mentioned by Participants)

 

Fig. 3. Summary of findings on users’ knowledge of Google’s business (taken from Lewandowski, Kerkmann, et al., 2017)

 

Table 2. Different Elements on the SERPs Used in this Study

 

Table 3. Query and Results Page Criteria in the Tasks Used

• Please imagine it is time to do your taxes and you are looking for help on the Internet. You find the search engine results page shown below. Please look to see whether there are any advertisements on this page. If there are, please mark every single one of them.

• On the following results page, you see the results for “buy glasses.” Please select all areas for which Google was not paid.

• Assume you are searching on Google for information about the career of designer Vivienne Westwood. You get the following search engine results page. Please look to see whether there are any advertisements on this page. If there are, please mark every single one of them.

• You searched for “buy laptop,” because you would like to buy such a computer. You get the following search engine results page. Please select all areas for which Google was not paid.

The result can be summarized as follows: All tasks taken together, only 1.3 percent of participants were able to mark all areas correctly, i.e., they either marked at least one advertisement as an organic result or vice versa. We also found that only 9.6 percent had all their identifications correct but did not mark all results they were required to mark.

Regarding the individual tasks (Table 4), we can see that for none of the screenshots given were more than 35% of users able to mark all areas correctly. We can further see that the more complex the results presentation gets (from Q8 being the easiest to Q11 being the most complex), the less the percentage of users who are able to mark all areas correctly.

Depending on the task, between 32.3% and 66.3% of users were able to mark only results that they were required to mark (either advertisements or organic results), but failed to mark all results of that type. For instance, participants marked only advertisements but 
not all advertisements shown on the SERP. A considerable share of users either only marked ads that were shown on the right-hand side or above the list of organic results.

All in all, the results show that the overwhelming majority of users is not well able to distinguish advertisements from organic results on Google’s search engine results pages. For a more detailed analysis of the results, including correlations between the results from the survey (e.g., age, education, and users’ self-reported knowledge on Google’s business model) and task performance, see Lewandowski, Kerkmann, et al. (2017).

 

6. EXPERIMENT

In this section, we report on the results of an online experiment that have previously been published in Lewandowski, Sünkler, et al. (2017). The stimulus material was two screenshots of a Google search engine results page (Figure 5). Both had the same results and the same layout, the only difference being one having the first two results labelled as ads (yellow shading, additional info button), while the other version only had a list of organic results. Note that in both versions, we 

 

Fig. 4. Screenshots of the SERPs used in the study

 

Table 4. Results for all Tasks

 

only used organic results, i.e., the ads shown in the experimental condition are actually organic results, only being labelled as ads.

The participants were given the following search task: “Imagine you participate in a cooking competition where you should prepare fresh calamari. Your search query is “calamari recipe.” Which result(s) would you click on spontaneously?” [translated from German].

In our online experimental setting, participants were asked to mark the results they would select, i.e., we did not measure actual clicks, but users labelled results as relevant. This allowed users to mark more than one result. Each user was randomly assigned either to the experimental condition (ads) or the control condition (no ads). In the following, we analyze the clicks on the first two results (the ads in the experimental condition and the first two organic results in the control condition, respectively).

First, we looked at relationships between users’ knowledge of whether it is possible to buy screen real estate on the search engine results pages and these users selecting ads. There are three groups in this case: Those who say that it is possible (correct answer), those who say it is not possible (incorrect), and those who say they do not know. Looking at the selection behavior in the two conditions, we do not find significant differences related to users’ knowledge (Table 5). While users in the control condition select the first two results more frequently than users in the experimental condition, there are no significant differences between groups within the conditions.

We found, however, significant results when it comes to users’ knowledge of whether it is possible to distinguish between paid advertisements and organic results on the search engine results pages (Table 6). In the experimental group, users who say that this is possible select the first result significantly more often. However, the group saying they do not know select the top results even more often. As expected, there are no 

 

Fig. 5. Screenshots used as stimulus material (left: organic results only; right: including ads)

 

Table 5. Results for the Question “Is it possible to pay Google for preferably listing one’s company on the search results pages, as an answer to a search query?”

 

Table 6. Results for the Question “Is it possible to distinguish between paid advertisements and unpaid results on Google’s search engine results pages?”+

 

significant differences in the control condition. Especially, the result that users who say it is possible to distinguish between the results types select the top result (in this case, an ad) more often seems counterintuitive. However, this may be explained by users who say this is not possible but still select the results they think are not advertisements.

The third knowledge-based question related to users’ knowledge of how Google makes money from its search engine. We classified the answers into four groups: Correct answer (advertising), incorrect answer, partly correct answer (where advertising was mentioned, but other incorrect sources of revenue, as well), and “don’t know” (where users admitted they did not know). We found significant differences in the selection behavior of the different groups only in the control conditions (Table 7). Users who know how Google makes money choose the first position significantly more often than users without that knowledge. A likely explanation is that these users notice that there are no ads on the page and therefore regard the first result as trustworthy. Next, we turn to performance-based measures, i.e. we distinguish between users who proved in the taskbased part of the study that they were able to identify ads consistently and those who were not able to do so. We found significant differences between selection behavior on the first two results between users who proved themselves to be able to distinguish between organic results and ads and those who were not able to do so (Table 8). The latter chose advertisements significantly more often (40.3% vs. 21.6% for the first results, and 28.3% vs. 13.7%, respectively). The numbers are quite impressive: Users who are not able to distinguish between the two results types choose ads around twice as often as users who are able to recognize the ads. As 

 

Table 7. Results for the Question “How does Google generate its revenue?”

 

Table 8. Actual Performance (Marking Ads)

 

expected, there are no significant differences in the selection behavior in the control condition.

 

7. DISCUSSION

The results presented show that the overwhelming majority of users are not well able to distinguish advertisements from organic results on Google’s search engine results pages. The task-based study showed that merely 1.3 percent of participants were able to mark all areas correctly, i.e., they either marked at least one advertisement as an organic result or vice versa. We also found that only 9.6 percent had all their identifications correct but did not mark all results they were required to mark. From the survey, we found that there is much confusion about Google’s business model and the way search engine advertising works among users. Results from the online experiment show that users not knowledgeable of the distinction between ads and organic results click on advertisements approximately twice as often.

It should be stressed that ads can be helpful to satisfy a user’s information need, and therefore, the distinction between organic results and ads may not be relevant to users seeking information in areas where ads can serve as pointers to content that satisfies their information needs. We should keep in mind, however, that while it is not necessarily a bad thing clicking an ad, it is a bad thing clicking an ad and not knowing having done so.

Our study revealed some surprising results, namely that users’ self-reported knowledge on whether it is possible to pay for being shown on Google’s SERPs did 
not affect their selection behavior in the experiment. We have to note, however, that we used self-reportedmeasures, i.e., we do not know whether users saying that they know that it is possible to pay Google for being shown on the SERPs actually know how to spot paid results. Those who say it is possible to distinguish between ads and organic results select ads more often. The likely reason is that they do so on purpose, as ads can be relevant to a search query (B. J. Jansen, 2007; 
Lewandowski, Kerkmann, et al., 2017). Users who know how Google generates its revenues selected the first results more frequently in the control condition, which could be related to them trusting the search engine’s ranking (Pan et al., 2007). Maybe even more interesting, the performance measures based on distinguishing users into a group that actually proved to be able to distinguish between ads and organic results in a variety of tasks, and a group that was not able to do so, show a clear (and in some ways contradictory) result from the other results from the experiment. They clicked on the ads approximately twice as often as the knowledgeable group. Through the experimental design of our study, we can rule out different relevance judgments as a reason for this, as users in both conditions saw the same results.

Our study has some limitations: Obviously, our results only hold true for the German online population. We can assume, however, that while there may be differences between users in different countries, the general finding will still hold true. Furthermore, we only used the desktop versions of the SERPs in the task-based study and the experiment. As smaller mobile screens do not allow for as many results to be displayed in the area “above the fold,” we can assume that users in this setting will select ads even more often than on the desktop. Further studies should focus on mobile user behavior, as by now, more than 50% of all queries come from mobile devices (Sterling, 2016).

Another point to consider is that Google’s results presentation has changed since the study was conducted, and can be expected to keep changing. We can, however, assume that the new results presentations used since then further blurred the line between organic results and advertisements (Edelman, 2014). We can go as far as to question whether the division between the two results types is at all fruitful, or whether due to the confusing mixture of paid-for and not paidfor results on the SERPs, we should see SERPs as comparable to infomercials (Lewandowski, 2016).

The results presented in this paper have implications for information science theory: First, based on the finding that a large part of users take search-based ads as search results not to be distinguished from organic results, i.e., they read results presentations containing the two results types as one single list, information retrieval effectiveness evaluation should adjust to this behavior. Studies investigating Web search engines should consider both results types and model user behavior respectively. In that way, information retrieval evaluation could become more realistic. The same applies to studies analyzing selection behavior, which by now have solely focused on the organic results (e.g., Goel et al., 2010; Keane, O’Brien, & Smyth, 2008). Such studies would considerably profit from measuring click-through rates for both results types.

Second, if the goal of information science is seen as enabling users to become better informed (Buckland, 2012, p. 5), then information science and practice should on the one hand focus on increasing users’ information literacy regarding search engines. Efforts on increasing users’ information literacy often focus on specialized information sources, and less on the tools hat users are using on a daily basis. It is time for information literacy researchers and practitioners to focus on that blind spot.

On the other hand, information science should focus on building systems that allow for a fair balance between search engine vendors’ interests in making money from advertisements and the search engine users’ interest in knowing whether a result shown on a SERP has been paid for or not.

Third, models of search engine advertising (J. Jansen, 2011, p. 206) need to consider users confusing ads with organic results. Also, the more general information seeking models should be expanded, as they are not concerned with specifics of users’ results selection. They are based on the presumption that users are competent in selecting results from a results presentation of a search system, or that (lack of) competence is not an issue at all. Search process models like Ellis’s (1989) and Marchionini’s (1995) do not differentiate at the results selection stage. Results selection is regarded as merely choosing relevant results from a list that is 
constituted of more relevant and less relevant results. The classic information retrieval model (see Saracevic, 2016) assumes that a system produces one result set from which the user selects relevant results. It does not consider the user beyond making selections and judging on the relevance of the selected results.

Judgments on results quality and, as part of it, credibility and trustworthiness, are regarded as being made after the results selection. This approach is on one hand limited because users already judge the (presumed) quality of a results document based on the result description (“snippet”) on the SERP (see Lewandowski, 2008). On the other, a maybe even more severe limitation is that we can regard inadequately labelled ads taken as organic results by a user as tricking the user into trusting a search engine based on false assumptions.

Models of results selection either focus on click distributions or on “information scent,” i.e., indicators for the attractiveness of a search result based on a result description / snippet. Card et al. (2001, p. 499) define information scent as “the (imperfect) perception of the value, cost, or access path of information sources obtained from proximal cues, such as WWW links.” Information scent is certainly a concept that also applies to search-based ads. However, the question of how ads 
and organic results compete in regards to information scent within the same results presentation (SERPs) has not yet been thoroughly researched.

Our results have implications for search engine design and regulation. The results call for a clear labelling of advertisements on the search engine results pages. There has been a discussion on how ads should be labelled for at least 15 years now (Sullivan, 2013b), which has, however, not been based on a proper empirical basis. Our study contributes to filling this gap. However, it should be regarded as only a first step. Further research is needed, especially related to further results types, such as Universal Search results, which can be either ads or organic results.

 

8. CONCLUSION

In this paper, we presented a large-scale study on users’ understanding of search-based advertising, based on (1) a survey, (2) a task-based user study, and (3) an 
online experiment. With a representative sample of 1,000 users from the German online population, the results are not only of high validity but also allow for some generalizations. The findings show that users lack an understanding of Google’s business model and the workings of search-based advertising. From the survey, we can see that 42% of the German Internet users self-report that they either do not know that it is possible to pay Google for preferred listings for one’s company on the SERPs or do not know how to distinguish between organic results and ads. In the taskbased user study, we found that only 1.3 percent of participants were able to mark all areas correctly. 9.6 percent had all their identifications correct but did not mark all results they were required to mark. For none of the screenshots given were more than 35% of users able to mark all areas correctly. In the experiment, we found significant differences between selection behavior on the first two results between users who proved themselves to be able to distinguish between organic results and ads and those who were not able to do so. Users who are not able to distinguish between the two results types choose ads around twice as often as users who are able to recognize the ads.

These results show that users trust in Google may be misplaced. If users cannot distinguish between organic results and ads, they will see ads as trustworthy content, whereas, while ads may be relevant to a query, the messages communicated by ads are always biased towards the interests of the advertiser. From our study, we can draw the conclusion that on the SERPs, Google does not make its money simply from advertisements but from advertisements that are in large part confused with organic results by the users. The blurred line between the presentation of snippets for organic results and for ads can even be found in Google’s advertising policies: Google does not allow “ads or extensions that are inconsistent with the clear and informational presentation style of the Google Search results” (Google, 2017).

It is obvious that our results call for regulation. Our data provide a basis for new argumentation. While our study focused on Google as the most popular Web search engine, search-based advertising is much more widely spread than just general-purpose search engines like Google or Bing. For instance, this type of advertising can now be found at sites like Amazon and Ebay, as well. The list of search sites that have been warned by the Federal Trade Commission in the United States (Sullivan, 2013b) is quite impressive and contains names like Yelp, TripAdvisor, and Yahoo Travel. Further research should investigate the general effect search-based advertising has in different contexts.