Other works by this author can be found at http://bibliomining.com/nicholson/

How Much of It is Real? Analysis of Paid Placement in Web Search Engine Results

Scott Nicholson (scott@scottnicholson.com), Assistant Professor

Syracuse University School of Information Studies

4-206 Center for Science and Technology

Syracuse, NY 13244

http://www.scottnicholson.com

Tito Sierra, Digital Technologies Development Librarian

North Carolina State University Libraries

Raleigh, NC

U. Yeliz Eseryel, Ji-Hong Park, Philip Barkow, Erika J. Pozo, Jane Ward

Syracuse University School of Information Studies

Syracuse, NY

Abstract

Most Web search tools integrate sponsored results with results from their internal editorial database in providing results to users. The goal of this research is to get a better idea of how much of the screen real estate displays “real” editorial results as compared to sponsored results. The overall average results are that 40% of all results presented on the first screen are “real” results, and when the entire first Web page is considered, 67% of the results are non-sponsored results. For general search tools like Google, 56% of the first screen and 82% of the first Web page contain non-sponsored results. Other results include that query structure makes a significant difference in the percentage of non-sponsored results returned by a search. Similarly, the topic of the query can also have a significant effect on the percentage of sponsored results displayed by most Web search tools.

Introduction

Since 1996, there has been a growing trend toward reliance on the Internet for information needs. In order to find this information, many users start their exploration with a Web search tool. Most Web search tools are for-profit enterprises and use advertising to bring in enough income to allow them to continue offering their services for free to the public. Most search tools offer some type of targeted marketing where advertisements are connected to certain search terms. Some ads are placed on the top, side, and bottom of search results, while other ads are placed within the context of the “real” results. The result is that a searcher may be faced with more ads than actual editorial results. The goal of this research work is to gain a better understanding of the amount and type of advertising presented to search tool users.

This type of advertising is not necessarily bad or inappropriate. A user looking for products or services may have their information need met through this type of contextual advertising. This model of large eye-catching advertisements to support an information source has been employed for decades through our commercial telephone directories. The food and beverage industry sells logo space on tabletop displays and menus, which can draw the eye from the text listing of the products available. Product placement outside of commercials has become more rampant in television shows and radio personalities plug products during their normal monologues. This model of contextual advertising has therefore been used and accepted in other domains.

There are some concerns about this type of contextual advertising. In some cases, the advertisements look similar to or are indistinguishable from non-paid listings relevant to the search terms (Wouters, 2004). Users expect these tools to provide information that is the best match to their query; instead they receive sponsored results from the highest bidder. As search tools become publicly-owned companies, they have to answer to a profit-seeking ownership. Thus, the temptation to provide more contextual advertising and fewer non-paid results grows.

In order to understand to what level search tools are providing advertisements over non-paid results, the research team submitted a series of controlled queries to eight different search tools. We then determine the amount of paid placement in the first screen and entire first page of results and identify patterns based on topic, query formulation, and type of search tool.

Research Questions

In order to explore paid placement in Web search tools, the following research questions are used:

1. Over all of the queries and tools, what mixture of content is displayed on the screen? Do different search tools present different amounts of advertising for the same searches? How does examining only the first screen of results differ from viewing the entire first page?

2. How much advertising is displayed for different searching topics? Do academic search topics produce less advertising than commercial search topics? Are there differences in the common topical searching areas?

3. How much advertising is displayed for different query structures? Do longer queries produce less advertising?

4. Are there patterns of interest in the data across facets of searches and tools that lead to future research questions?

Definitions and Assumptions

The primary definition to be discussed is that of an “editorial listing,” or the “real” results from a search tool. A search tool sends out spidering programs to Web pages that return with words and other information located on that Web page. These terms are integrated into an inverted index and an editorial listing is created to represent that Web page. This index is used in conjunction with matching and weighting algorithms to determine the pages returned for a query. In some cases, the pages in this primary index are there through paid inclusion, where a Web site can pay for rapid consideration for inclusion into a Web search tool index. There is no way to discern between pages that are in the primary index because a Web exploration program (a.k.a. spider) discovered them or because of paid inclusion. Thus, all of these results are grouped into the category of “editorial listings” that are presented to a user purely because they are the most relevant items in the primary database.

The term “editorial listing,” used by others in the literature (Phillips, 2001, for example), does not imply that there were any decisions made by a human about the quality of the site. However, each search tool does have its own policies as to what pages to accept into the database. Most search tools will not index every page on the Internet; there are some basic quality standards that guide their selection. Many times, search tools will not index pages that are using tricks to raise their ranking, such as very small text or text that is the same color as the background. In addition, some tools do not index every page on a site, and index only those pages in the top levels of the site. Therefore, while the criteria a tool uses for page inclusion may not be as strict as a human’s selection, there is usually some type of editorial policy guiding what pages are indexed in the tool. The term “editorial listing” (or “real” result) reflects that these pages are part of the Web page database because they fit the editorial policies of the search tool, as automatically detected by a Web spider.

The other type of result provided to users is that of a sponsored advertisement. These listings are not visited by the traditional search tool spider, and the sponsor determines the content of the listing. Many times, sponsors pay to be listed when a user searches on specified terms, and the amount they are willing to pay determines their prominence. These listings may be graphical advertisements on the top or side of the screen or may be textual results placed within the context of the editorial results. Both of these are considered sponsored results, as they present choices to the user that represent other Web pages. The contrast in the current study is between the surrogates for pages chosen for relevancy, i.e. editorial listings, or surrogates chosen because someone paid the search tool to have them included, i.e. sponsored listings.

As with any search tool research, one assumption is that patterns found in Web search tool results are stable over time. This assumption is short-lived - as the search tools change, the results of the study will also change; this is the nature of Web research. The goal of this study is to provide patterns found from a snapshot of time, with hopes that these patterns will be useful in analyzing future situations.

Another assumption is derived from the measurement tool used for the study, which is based on one used by Nielsen and Tahir (2002). They used “screen real estate” as a measure in examining the content components of Web pages. The use of this measure is based on the assumption that the amount of screen real estate used to display results in a Web search tool is correlated to the amount of information provided by those results. This is not necessarily true in all cases. However, in this study this assumption allows us to discuss the concepts using a unit of measure that is comparable between tools – the percentage of a screen dedicated to a type of information. Throughout this paper, terms such as “amount of information” do not refer to relevance, the user’s perception of Web links, the number of links, or the content of those links. All of these possible measures represent areas for other research projects. In this paper, however, the measure used is the percentage of screen real estate.

Literature Review

There has been considerable discussion of paid placement in Web search engines in popular literature, but very little in academic literature. Trade articles and opinion pieces citing the dangers of paid placement for users are plentiful, though the extent to which paid placement influences results of Web search engines has not been comprehensively measured. This literature review focuses on the user issues related to paid placement in Web search tools such as bias, consumer awareness, and disclosure. Additionally, it highlights prior research supporting the methodology choices made in the present study.

User Issues

In their paper on bias and search engines, Mowshowitz and Kawaguchi (2002a) argue that bias is present when some items occur more frequently or prominently with respect to the norm, while others occur less frequently or prominently with respect to the norm. In their definition, bias may be introduced into a retrieval system at any stage of its operation. Sources of bias include the rules used for document inclusion, the way items are indexed, the manipulation of text in Web pages to increase retrieval, query formulation constraints or predispositions, and the search algorithms used to retrieve and rank items. Although Mowshowitz and Kawaguchi do not mention the role of paid placement explicitly, it is clear that sponsored advertising introduces bias in Web search engine results.

In a subsequent study, Mowshowitz and Kawaguchi (2002b) measure the bias present in fifteen commercial search engines using statistical analyses of variance in results returned. Bias values are computed by analyzing the top thirty URLs returned by each search engine for search terms across eight subject areas. The results suggest that their measure of bias can discriminate between search engines, but for most search engines bias does not depend on the subject domain searched, or the search terms used to represent that subject domain. The current research explores this finding to see if bias from sponsored placement depends upon the search topic.

Bhargava and Feng (2002) analyze the tension between benefits (to providers) and disutility (to users) inherent in search engine paid placement. An assumption in their analysis is that the perceived disutility to users, caused by search engine provider bias, can negatively impact search engine market share and user-based revenues. Their analysis leads them to develop a mathematical model for optimal design of a paid placement strategy.

However, not everyone agrees that paid placement inherently creates a disutility to users of search tools. Bill Gross, whose company Idealab founded Overture, recalls telling his Idealab colleagues in 1997 that “the best way to clean up search results was to use money as a filter” (Hansell, 2001). Gross finds that, at the time, search engines return a disproportionate number of results from porn Web sites; therefore, paid placement could operate as a filtering mechanism. Gross’ idea eventually led to the creation of the GoTo.com Web search engine where advertisers would bid for top placement in search results. The bid for placement model that GoTo.com helped popularize is akin to the Yellow Pages model, where the advertisers willing to pay the most get the most prominent placement. Yellow Pages combine textual listings of telephone numbers for companies with larger graphical advertisements for some of those companies. The Yellow Pages model suggests that some types of commercial searches (such as searches for products or services) can benefit from paid placement.

A study by Graham and Metaxas (2003) suggests that college students rely heavily on the Internet for their research needs. Participants in the study completed a survey of six questions conducted over e-mail during the 2000-2001 school year. The results of the study reveal students’ extreme confidence in search engines as a research tool, despite their having little awareness of how search engines select results. Additionally, the study suggests that students remain faithful to one search engine, even if it does not immediately provide the answers sought. Although not mentioned in the study, these findings also suggest a need to understand what impact paid placement has on the use of search engines in academia. The present study of paid placement considers academic and non-academic queries separately.

An ethnographic study commissioned by Consumer WebWatch in 2003 shows that participants had little understanding of how search engines ranked pages, and many were surprised to learn many search engines employed paid placement strategies (Marable, 2003). The study asked seventeen participants from varying demographic backgrounds to perform two online searches on five pre-assigned search sites. A total of fifteen major search and navigation Web sites were tested in the study. This study suggests a consumer awareness problem with regards to paid placement in search engines, and recommends that search engines take steps to fully disclose which results are ranked higher because of paid placement.

This same organization, Consumer Web Watch, subsequently released a report looking at the policies of the search tools regarding sponsored listings (Wouters, 2004). Their report shows that search tools do not do an adequate job of informing customers about sponsorship in their results in their help files or policy documents. Moreover, many tools do not strive to provide a significant separation between sponsored results and editorial results. Meta search tools proved to be a consistent offender by taking sponsored results from different tools and mixing them in with editorial results.

Two screen captures from 2003 demonstrate this concern. Figure 1 is from Overture.com, which allows Web site marketers to bid for placement. The highest bid receives first ranking and after all paid rankings have been displayed, Overture displays editorial results. The search topic was “basketball,” and all of these listings are properly identified as sponsored listings. This is appropriate behavior for a search tool, as their advertising is labeled.

Figure 1. Screen shot from Overture (2003)

Figure 2 shows a screen shot taken the same day from Excite, which is a meta-search tool. Users are led to believe by the top line that Excite has combined results from all of the search tools into one listing.

Figure 2. Screen shot from Excite (2003)

The same paid results in Overture are listed in Excite as the top-ranked results. The problem is that these results are not labeled as sponsored. In fact, Overture owned Excite and was using it as another way to distribute their results. Users have no way of knowing that these top results are there simply because they paid top dollar for those spots. This problem was the impetus behind the study. As an aside, many search tools have improved their labeling of advertisements since these screen captures were taken.

It is possible that a search tool has both an editorial listing and a sponsored listing for the same commercial Web site. For this study, these would be counted as different types of listings, as the methods by which they were ranked and listed were different. We used contextual clues and information from the pay-for-placement tools, as in Figure 1 and Figure 2, to identify sponsored links that were not labeled. We were cautious in labeling the results this way, so the results related to the amount of unlabeled sponsorship should be taken as the minimum for that measure.

In 2001, the non-profit group Commercial Alert filed a complaint to the FCC requesting that it investigate whether seven major search engine companies were violating existing advertising disclosure laws in their use of paid placement and paid inclusion. In response, the FCC stated that it would not take formal action against the seven search engines, but would send a letter to each of the seven search engine companies outlining the need for clear and conspicuous disclosures of paid placement. The FCC letter to search engine companies outlined three specific recommendations: 1. paid ranking search results are clearly distinguished from non-paid results, 2. the use of paid inclusion is explained, and 3. no affirmative statement is made that might mislead consumers as to the basis on which a search result is generated (FTC, 2001).

Introna and Nissenbaum (2000) examined the political issues raised by search engines from a qualitative perspective, addressing the specific issue of whether the market mechanism can serve to correct search engine bias. They argued that since most users of the Web lack critical information about search engine alternatives and little understanding of how search engines work, users are not truly exercising free choice. Their recommendations include the need for full and truthful disclosure on the part of search engines regarding the rules governing indexing, searching, and prioritization, and consideration of public support for egalitarian and inclusive non-commercial search tools.

Methodological Review

Prior research about Web searching and use was examined to determine what topics were selected for searching, what types of queries were formed, the structure of those queries, which tools were selected, and how the results were judged. These studies served as inspiration to create a study that is representative of the most common searches of the Web.

Query and Tool Selection

A 2002 study by Whitmire using the Biglan model of disciplinary differences distinguish information-seeking behavior patterns among various academic disciplines. Data for their study were obtained from a 1996 College Student Experiences Questionnaire and grouped students according to the three Biglan dimensions (hard-soft, pure-applied, life-non-life). Although this study aims to understand how different disciplines compare with one another in their use of library resources, there may be value in applying the Biglan framework specifically to academic use of Web search tools. In the present study, these dimensions are used to formulate six different academic query topics, found in Table 1.

Spink, Wolfram, Jansen and Saracevic studied a log of over one million Web queries by users of the Excite search engine to understand the general public’s practices and choices in Web searching (2001). A product of this study is a classification of 11 major search subject categories. In a subsequent analysis of the Excite data set, Ross and Wolfram (2000) categorized the 1,054 most frequently co-occurring term pairs into one or more of 30 subject areas. The five most popular subject categories in the Spink et al. study have been used as the basis for search categories representing non-academic users for the current study. These categories are summarized in “Non-academic” column heading in Table 1.

A query log analysis by Silverstein, Marais, Henzinger, and Moricz of approximately 1 billion queries submitted to the Altavista Web search engine found that the public used an average of 2.35 terms per search query (1999). 20.6% of AltaVista queries were submitted with no terms (an empty query), 25.8% had one term only, 26% had two terms, and 15% had three terms. The Spink et al. Excite study cited earlier found that the public used an average of 2.4 terms per distinct query. Excluding the 9.7% of queries submitted with zero terms, 26.6% of Excite queries had one term only, 31.5% had two terms, and 18.2% had three terms. In the current study, we test the impact of paid placement on one, two, and three term queries separately. Additionally, the single-term query tested for each category is retested with an additional related term to test how the impact of paid placement changes with search terms that are not phrases. The four different categories for query term length are represented as column headings in Table 1.

We differentiate between academic and non-academic searches in the current study. For each class of search, we use a single query to represent the intersection of subject category (e.g. “engineering,” “people, places, things”) and query term lengths (e.g. “1 query term,” “2 query terms”). These queries appear in the boxes in Table 1 below. We used two different methods to select specific queries for academic and non-academic searches.

To derive queries for each of the six academic categories, we selected academic subject terms appearing in course titles in the 2003-2004 Syracuse University undergraduate course catalog. These queries are meant to be representative of possible subject-specific queries submitted by undergraduate students, but not necessarily representative of the most popular academic queries submitted by students.

For non-academic queries, known popular search queries were selected this study. The primary resource was a set of yearly statistical reports published by the Lycos search engine based on their ongoing “The Lycos Top 50” report (http://50.lycos.com/). As the name implies, The Lycos Top 50 is a list of the fifty most popular query terms submitted by Lycos users for a given time period. We used the Lycos top 50 early reports for 2000, 2001, and 2002 for our query selection process. To be included in our test list, the popular queries had to appear in the top 100 for at least 2 of the 3 years surveyed, to minimize the inclusion of short-lived topics of interest. Lycos excludes queries identified as “prurient” (e.g., 4-letter words, pornography terms) from its lists. To find specific queries for the “sex, pornography, preferences” category, we constructed queries based on the most popular query terms and query term pairs identified by the Spink et al. Excite study (2001).

Academic	1 term	w/ additional term	2 term phrase	3+ term phrase
Physical sciences	genetics	statistics genetics	organic chemistry	solar system astronomy
Engineering	propulsion	momentum propulsion	soil mechanics	computer aided design
Humanities	Writing	literature writing	architectural history	cultural theories of representation
Business	securities	securities privatization	consumer behavior	supply chain management
Social sciences	psychology	psychology marriage	behavior disorders	criminal justice system
Education	pedagogy	pedagogy religion	teacher development	adult literacy education
Non-academic
Entertainment, recreation	baseball	baseball women	star wars	buffy the vampire slayer
Sex, pornography, preferences	sex	sex disease	xxx pics	free nude pictures
Commerce, travel, employment, economy	taxes	taxes bankruptcy	mortgage rates	world trade center
Computers, the Internet	napster	napster mp3	final fantasy	grand theft auto
Health, the sciences	diabetes	diabetes glaucoma	atkins diet	science fair projects
People, places, things	paris	paris london	britney spears	martin luther king

Table 1. Matrix of query terms used

These queries are used with three general Web search tools, three meta search tools, and two search tools primarily based on paid listings.

Sullivan summarizes search engine popularity in terms of “Audience Reach” and “Time Spent on Site” based on data from a January 2003 Nielsen NetRatings search-specific report (Sullivan, 2003b). Based on audience reach, the three most popular Web search tools are Google, Yahoo, and MSN, accounting for a combined total of 86% audience reach.

Meta search tools are those that collect results from several different search tools in response to a query. Wouters (2004) found that meta search tools are the worst offenders in displaying sponsored results that are not appropriately labeled or disclosed. Dogpile, Mamma, and ixquick are used for the meta search tests. Dogpile is identified as one of the “Big Four” meta search engines (Sherman, 2002); however, all of the “Big Four” metasearch engines are owned by the same parent company, InfoSpace, and return similar results for the same search. Therefore, two other meta search tools listed at Searchenginewatch.com were selected to supplement Dogpile.

We also tested our matrix of user queries against Overture and FindWhat.com, which provide paid listings for many other Web search tools.

Category	Name	URL
Paid Listings Provider	1. Overture	http://www.overture.com
Paid Listings Provider	2. FindWhat.com	http://www.findwhat.com
General Web search	3. Google	http://www.google.com
	4. Yahoo! Search	http://www.yahoo.com
	5. MSN Search	http://www.msn.com
Meta search	6. Dogpile	http://www.dogpile.com
	7. Mamma	http://www.mamma.com
	8. ixquick	http://www.ixquick.com

Table 2. Web search tools tested

Methods for Analysis of Results from Previous Literature

In a limited study of paid placement in meta-search engines, Danny Sullivan of SearchEngineWatch.com (2003) quantified paid placement in terms of the percentage of links on the first page of results that were “paid links.” Sullivan tested for paid links by examining the results for the single query term “canada” across eight well-known meta search engines. The range of paid links ranged from 0% to 86% depending on the search engine. Sullivan admits that the results he observed could vary, since only a single query term and time of day were tested. The problem with using a number of links is that a link can take many forms – a banner advertisement, a small listing, a colored and prominent listing with the words “sponsored link” at the bottom of the page. For this study, the goal was to measure what the user sees on the screen, which is more than just the link.

In their study of newspaper Web sites, Dewan, Freimer and Zhang quantified the amount of advertising in terms of the percentage of screen real estate devoted to advertising (2002). The percentage of advertising space was calculated as a percentage of an 800x600 pixel screen, as that was the most common resolution when this study was designed. A similar screen real estate approach is used by Nielsen and Tahir, in their study of large-scale Web site homepages, to calculate the percentage of homepage real estate devoted to navigation, advertising, site identity, etc. (2002). In our study we have selected the “screen real estate” approach to calculate the impact of advertising over the percentage of links approach, as it is a more accurate measure of the visual impact of advertising on users.

There are two general types of advertising commonly used in Web search tools. The first type, Non-integrated Advertisements, comprises paid advertisements that are presented differently from editorial listings. These are listings for which no payment for preferential placement has occurred and are clearly separated from the editorial listings on the search results page. Non-integrated Advertisement can be either text-based or graphical in nature. The second, Integrated Paid Listings, comprises search result listings for which payment for preferential placement has occurred.

Integrated Paid Listings can be conspicuous or inconspicuous, depending on whether or not they resemble editorial listings, or depending on the extent to which they clearly labeled as paid listings. In this study, we used the pay-for-placement search tools such as Overture and Findwhat to identify inconspicuous paid listings. As in the example in Figures 1 and 2, it is clear when a tool has taken listings from one of these sponsored tools and listed them as editorial listing. It is probable that there are listings we were not able to identify as unlabeled paid listings when more devious methods of accepting payment for listing were employed. Therefore, this measure should be taken as a minimum number of listings, which may be actually be higher.

The FCC has recognized the use of ambiguous terminology in the labeling of paid listings (2002). In the present