Google outlines risks of exposing its search index, rankings, and live results

Google’s head of Search warned a federal court that forcing the company to share its search index, ranking data, and live results with competitors would cause “immediate and irreparable harm” to Google, its users, and the open web.

The warning appears in a filed affidavit from Elizabeth Reid, Google’s vice president and head of Search, submitted with Google’s motion to pause key antitrust remedies while it appeals the final judgment in the DOJ search monopoly case.

The filing spells out what Google sees as its most sensitive Search assets and why sharing them would expose proprietary systems, enable reverse engineering, and fuel spam.

Disclosure of Google’s web search index

The fight: Section IV of the final judgment would force Google to give “qualified competitors” a one-time dump of its core web index data at marginal cost. That data would include:

• Every URL in Google’s web search index
• A DocID-to-URL map
• Crawl timing data
• Spam scores
• Device-type flags

Google’s argument: This would hand competitors the output and the accumulated insight of more than 25 years of indexing work.

Reid described the index as the product of proprietary crawling, annotation, and tiering systems that decide which pages enter Google Search:

• “The selection of webpages in Google’s search index is the result of more than twenty-five years of sustained investments and exhaustive engineering efforts.”

She warned that simply knowing which URLs Google indexes would allow rivals to skip large portions of crawling and analysis altogether:

• “Receiving the list of URLs in Google’s index will enable Qualified Competitors to forgo crawling and analyzing the larger web, and to instead focus their efforts on crawling only the fraction of pages Google has included in its index.”

Metadata such as crawl frequency would reveal how Google prioritizes freshness and demand, she added:

• “Information regarding Google’s crawl schedule will provide rivals with insight into Google’s proprietary freshness signals and index tiering structure.”

Included in the affidavit is this image, “Google’s Web Crawling and Indexing Process: The Results,” showing that Google labels the great majority of webpages as “Spam, Duplicates, & Low Quality Pages.”

• Google has crawled a redacted number of pages in the trillions. As of 2020, Google’s index contained roughly 400 billion documents, according to testimony from Pandu Nayak, a Google executive.

Risk of spam, abuse, and reputational damage

The concern: Google argues that exposing spam scores, even indirectly, would weaken its ability to fight webspam.

Effective spam fighting depends on secrecy, Reid stressed:

• “Fighting spam depends on obscurity, as external knowledge of spam-fighting mechanisms or signals eliminates the value of those mechanisms and signals.”

If spam scores leaked or were breached, bad actors could use them to bypass Google’s defenses, Reid warned:

• “Spammers … could bypass Google’s spam detection technologies and hamstring Google in its efforts to combat spam.”

That would push more low-quality and misleading content into search results, with users ultimately blaming Google:

• “The compelled disclosures are likely to cause more spam and misleading content to surface in response to user queries, compromising user safety and undermining Google’s reputation as a trustworthy search engine.”

Disclosure of user-side search data (Glue and RankEmbed)

What the judgment requires: Ongoing sharing of “user-side data” used to run Google’s Glue and RankEmbed models. Reid says that data includes:

• Queries
• Location
• Time of search
• Clicks, hovers, and other interactions
• Every result and search feature shown, and their order

Glue captures 13 months of U.S. search logs, according to the affidavit.

Google’s argument: This would amount to a massive, ongoing disclosure of Google’s ranking output at scale.

• “The disclosure of Glue training data amounts to the disclosure of Google’s intellectual property, because it reveals the output of Google’s Search technologies in response to every query issued by a user located in the United States over a 13-month period.”

She also warned that the data could be reused directly.

• “Qualified Competitors could also readily use the disclosed Glue and RankEmbed data as training data for a large language model.”

On privacy, Reid emphasizes that Google would not control the final anonymization decisions.

• “Google will not have final decision-making authority over the anonymization and privacy-enhancing techniques to be applied to the user data before it is shared.”

Users would still hold Google responsible for any fallout, Reid predicted.

• “Google users are nonetheless likely to fault Google for any privacy or security issues that arise from the data disclosures.”

Syndication of Google’s search results and features

What’s required: Section V would force Google to license and syndicate core search outputs to competitors for up to five years, including:

• Organic web results (“ten blue links”)
• Query rewriting
• Local, Maps, Images, Video, and Knowledge Panels

Google’s warning: This would expose the live output of its search systems to competitors—and beyond.

• “The search results and features required to be syndicated to Qualified Competitors are the product of decades of sustained engineering effort and innovation and many billions of dollars of investment.”

Even with contractual limits, Google would lose control, Reid said:

• “Google does not have the ability (as it does in the ordinary course) to decline to syndicate to a Qualified Competitor.”

Competitors could store, analyze, or leak the data — and that third parties could scrape it as well, Reid warned.

• “Any third party could ‘scrape’ the syndicated results and features from Qualified Competitors’ sites and thereby also avail themselves of Google’s results and features.”

The document. Read it here.

• What it is: Affidavit of Elizabeth Reid (Document #1471, Attachment #2)
• Filed: Jan. 16 at 3:46 p.m. ET
• Case: United States of America v. Google LLC, No. 1:20-cv-03010 (D.D.C.)
• Purpose: Supports Google’s motion to partially stay antitrust remedies pending appeal

Reid testified previously at the remedies hearing and said the affidavit reflects her personal knowledge as the executive responsible for all of Google Search.