{"id":7104,"date":"2026-04-24T21:06:36","date_gmt":"2026-04-24T13:06:36","guid":{"rendered":"http:\/\/longzhuplatform.com\/?p=7104"},"modified":"2026-04-24T21:06:36","modified_gmt":"2026-04-24T13:06:36","slug":"googles-robots-txt-docs-expand-deep-links-get-rules-eu-steps-in-seo-pulse-via-sejournal-mattgsouthern","status":"publish","type":"post","link":"http:\/\/longzhuplatform.com\/?p=7104","title":{"rendered":"Google\u2019s Robots.txt Docs Expand, Deep Links Get Rules, EU Steps In \u2013 SEO Pulse via @sejournal, @MattGSouthern"},"content":{"rendered":"<p><\/p> <div id=\"narrow-cont\"> <p>Welcome to the week\u2019s Pulse: updates affect how deep links appear in your snippets, how your robots.txt gets parsed, how agentic features work in Search, and how the EU\u2019s data-sharing rules apply to AI chatbots.<\/p> <p>Here\u2019s what matters for you and your work.<\/p> <h2>Google Lists Best Practices For Read More Deep Links<\/h2> <p>Google updated its snippet documentation with a new section on \u201cRead more\u201d deep links in Search results. The documentation lists three best practices that can increase the likelihood of these links appearing.<\/p> <p><strong>Key facts:<\/strong> Content must be immediately visible to a human on page load, and content hidden behind expandable sections or tabbed interfaces can reduce the likelihood of these links appearing. Sections should use H2 or H3 headings. The snippet text needs to match the content that appears on the page, and pages with content loaded after scrolling or interaction may further reduce the likelihood.<\/p> <h3>Why This Matters<\/h3> <p>The three practices are the first specific guidance Google has published on this feature. Sites using expandable FAQ sections, tabbed product detail areas, or scroll-triggered content for core information may see fewer deep links in their snippets compared with sites that render the same content on page load.<\/p> <p>The guidance matches a pattern Google has applied to other Search features. Content that renders without user interaction is more likely to appear in enhanced display.<\/p> <p>Slobodan Mani\u0107, founder of No Hacks, made a related observation on LinkedIn:<\/p> <blockquote> <p>\u201cThe documentation is framed around one snippet behavior (read more deep links in search results), but the language Google chose reads as a general preference. \u2018Content immediately visible to a human\u2019 is the structural instruction, not a read-more-specific tip.\u201d<\/p> <\/blockquote> <p>Mani\u0107\u2019s point extends his April 16 IMHO interview with Managing Editor Shelley Walsh, where he argued that most websites are structurally broken for AI agents. He argues that search crawlers and AI agents now face the same structural problem, and the audit is the same for both.<\/p> <p>For existing pages, the audit question is whether key information is contained within a click-to-expand element. If a page already has a \u201cRead more\u201d deep link for one section, that section\u2019s structure serves as a guide to what works. For other sections on the same page, replicating that structure may also improve their chances.<\/p> <p>Google describes the guidance as best practices that can \u201cincrease the likelihood\u201d of deep links appearing. That hedging matters because this is not a list of requirements, and following all three may not guarantee the links appear.<\/p> <p>Read our full coverage: Google Lists Best Practices For Read More Deep Links<\/p> <h2>Google May Expand Its Robots.txt Unsupported Rules List<\/h2> <p>Google may add rules to its robots.txt documentation based on analysis of real-world data collected through HTTP Archive. Gary Illyes and Martin Splitt described the project on the latest Search Off the Record podcast.<\/p> <p><strong>Key facts:<\/strong> Google\u2019s team analyzed the most frequently unsupported rules in robots.txt files across millions of URLs indexed by the HTTP Archive. Illyes said the team plans to document the top 10 to 15 most-used unsupported rules beyond user-agent, allow, disallow, and sitemap. He also said the parser may expand the typos it accepts for disallow, though he did not commit to a timeline or name specific typos.<\/p> <h3>Why This Matters<\/h3> <p>If Google documents more unsupported directives, sites using custom or third-party rules will have clearer guidance on what Google ignores.<\/p> <p>Anyone maintaining a robots.txt file with rules beyond user-agent, allow, disallow, and sitemap should audit for directives that have never worked for Google. The HTTP Archive data is publicly queryable on BigQuery, so the same distribution Google used is available to anyone who wants to examine it.<\/p> <p>The typo tolerance is the more speculative part. Illyes\u2019 phrasing implies that the parser already accepts some misspellings of \u201cdisallow,\u201d and more may be honored over time. Audit any spelling variants now and correct them, rather than assuming they will be ignored.<\/p> <p>Read our full coverage: Google May Expand Unsupported Robots.txt Rules List<\/p> <h2>EU Proposes Google Share Search Data With Rivals And AI Chatbots<\/h2> <p>The European Commission sent preliminary findings proposing that Google share search data with rival search engines across the EU and EEA, including AI chatbots that qualify as online search engines under the DMA. The measures are not yet binding, with a public consultation open until May 1 and a final decision due by July 27.<\/p> <p><strong>Key facts:<\/strong> The proposal covers four data categories shared on fair, reasonable, and non-discriminatory terms. The categories are ranking, query, click, and view data. Eligibility extends to AI chatbot providers that meet the DMA\u2019s definition of online search engines. If the Commission maintains eligibility through the final decision, qualifying providers could gain access to anonymized Google Search data under the Commission\u2019s proposed terms.<\/p> <h3>Why This Matters<\/h3> <p>This proposal explicitly extends search-engine data-sharing eligibility to AI chatbots under the DMA. If the eligibility survives the consultation, the regulatory category of \u201csearch engine\u201d now includes products that most search marketing work has treated as a separate category.<\/p> <p>The consequences vary depending on where you operate. For sites optimizing for EU\/EEA visibility, the change could broaden the scope of where anonymized search signals flow. AI products competing with Google in that market could use the data to improve their retrieval and ranking systems, which could, in turn, affect which content they cite.<\/p> <p>Outside the EU, the direct regulatory effect is zero. The category definition is a different matter. How the Commission draws the line between \u201cAI chatbot\u201d and \u201cAI chatbot that qualifies as a search engine\u201d is likely to be cited in future proceedings.<\/p> <p>The eligibility question is the story to watch through May 1. If the Commission narrows the AI chatbot criteria in response to consultation feedback, the implications stay regulatory. If it holds the line, that would set a material precedent for how AI search is classified.<\/p> <p>Read our full coverage: Google May Have To Share Search Data With Rivals<\/p> <h2>Google Adds New Task-Based Search Features<\/h2> <p>Google introduced new Search features that continue its evolution toward task completion. Users can now track individual hotel price drops via a new toggle in Search, and Google is adding the ability to launch AI agents directly from AI Mode.<\/p> <p><strong>Key facts:<\/strong> Hotel price tracking is available globally through a toggle in the search bar. When prices drop for a tracked hotel, Google sends an email alert. The AI agent launched from AI Mode allows users to initiate tasks handled by AI within the search interface. Rose Yao, a Google Search product leader, posted about the features on X.<\/p> <h3>Why This Matters<\/h3> <p>Each task-based feature moves a process that previously started on another site into Google\u2019s own surface. Hotel price tracking has existed at the city level for months. Expansion to individual hotels adds a new signal that users can set inside Google rather than on hotel or aggregator sites.<\/p> <p>Direct-booking visibility depends on being inside Google\u2019s ecosystem. Sites relying on price-drop alerts as a return-trigger for users may see some of that engagement reallocated to Google\u2019s tracking UI. For hotel brands, this raises the stakes for ensuring individual hotel pages are fully populated in Google Business Profile and hotel feeds.<\/p> <p>On LinkedIn, Daniel Foley Carter connected the feature to a broader pattern:<\/p> <blockquote> <p>\u201cGoogle\u2019s AI overviews, AI mode and now in-frame functionality for SERP + SITE is just Google eating more and more into traffic opportunities. Everything Google told US not to do its doing itself. SPAM \/ LOW VALUE CONTENT \u2013 don\u2019t resummarise other peoples content \u2013 Google does it.\u201d<\/p> <\/blockquote> <p>The AI agent launch is more speculative. Google has not published detailed documentation explaining what kinds of tasks users can delegate or how sources get cited. The feature confirms that agentic search, described by Sundar Pichai as \u201csearch as an agent manager,\u201d is appearing incrementally in Search rather than as a single launch.<\/p> <p>Read Roger Montti\u2019s full coverage: Google Adds New Tasked-Based Search Features<\/p> <h2>Theme Of The Week: The Rules Are Getting Written<\/h2> <p>Each story this week spells out something that was previously implicit or underway.<\/p> <p>Google signaled plans to expand what its robots.txt documentation covers. The company listed specific practices that can increase the likelihood of \u201cRead more\u201d deep links appearing. The European Commission proposed measures that extend search-engine data-sharing eligibility to AI chatbots under the DMA. And task-based features that Sundar Pichai described in interviews are rolling out as toggles in the search bar.<\/p> <p>For your day-to-day, the ground gets firmer. Fewer questions are judgment calls. What does and doesn\u2019t qualify, what Google supports, and what counts as a search engine to a regulator are all getting written down. That works to your advantage when it means clearer audit criteria, and against you when \u201cwe weren\u2019t sure\u201d is no longer a defensible answer.<\/p> <p><strong>Top Stories Of The Week:<\/strong><\/p> <p><strong>More Resources:<\/strong><\/p> <hr\/> <p><em>Featured Image: [Photographer]\/Shutterstock<\/em><\/p> <\/div> <p>News,SEO Pulse#Googles #Robots.txt #Docs #Expand #Deep #Links #Rules #Steps #SEO #Pulse #sejournal #MattGSouthern1777035996<\/p> ","protected":false},"excerpt":{"rendered":"<p>Welcome to the week\u2019s Pulse: updates affect how deep links appear in your snippets, how your robots.txt gets parsed, how agentic features work in Search, and how the EU\u2019s data-sharing rules apply to AI chatbots. Here\u2019s what matters for you and your work. Google Lists Best Practices For Read More Deep Links Google updated its [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":7105,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[2980,7381,2648,179,1467,90,841,13081,560,80,97,517],"class_list":["post-7104","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-accessibility","tag-deep","tag-docs","tag-expand","tag-googles","tag-links","tag-mattgsouthern","tag-pulse","tag-robots-txt","tag-rules","tag-sejournal","tag-seo","tag-steps"],"acf":[],"_links":{"self":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/7104","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7104"}],"version-history":[{"count":0,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/7104\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/media\/7105"}],"wp:attachment":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7104"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7104"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7104"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}