An XML sitemap lists the URLs of a site to facilitate their discovery by search engines. Google reminds us in its documentation: this file helps with URL discovery, but does not guarantee either crawling or indexing. Measuring the gap between what a sitemap promises and what it actually produces allows for calibrating the technical effort to be dedicated to it.
XML Sitemap and HTML Sitemap: two files, two distinct functions
The confusion between XML sitemaps and HTML sitemaps persists. Their roles, their recipients, and their SEO impact differ in several ways.
| Criterion | XML Sitemap | HTML Sitemap |
|---|---|---|
| Main Recipient | Crawlers (Googlebot, Bingbot) | Human visitors |
| Format | .xml file placed at the root of the site | Web page accessible via the menu or footer |
| Content | List of URLs with metadata (last modified date, frequency) | Clickable tree structure of pages |
| Impact on Crawling | Signals URLs to prioritize for exploration | No direct effect on crawler behavior |
| Impact on User Experience | None (file not intended for visitors) | Aids navigation, especially on large sites |
| Maintenance | Automatable via CMS or pipeline | Often manual or semi-automatic |
The XML sitemap remains the reference format in SEO. The HTML sitemap can complement the internal linking, but it does not replace a properly structured XML file. On the sitemap page of Pimp Your Biz, we see a concrete example of an XML file listed by section, which illustrates the logic of segmentation by content type well.
Indexable URLs in the sitemap: the filter that changes the signal sent to Google
Listing all the URLs of a site in the sitemap is a common mistake. A file that contains 301 redirects, noindex pages, 404 errors, or parameter variants sends a mixed signal to crawlers. Google then has to sort out what is worth exploring.
Only canonical and indexable URLs belong in the sitemap. This filtering principle transforms the file into a reliable inventory rather than a technical dump.
URLs to systematically exclude
- Redirected pages (301, 302): they point to another URL, including them creates an unnecessary loop for the crawler
- Noindex pages: their presence in the sitemap contradicts the meta robots directive and generates a technical inconsistency
- URLs with sorting, filtering, or session parameters: they duplicate the content of a canonical page without added value for indexing
- Pages with 404 or 410 errors: they waste crawl budget on non-existent resources
An e-commerce site with thousands of product listings illustrates the problem well. Without filtering, the sitemap can contain the same product multiple times under different URLs (sorted by price, color, size). The crawler then explores dozens of variants instead of focusing on the canonical page.
Automating sitemap generation to avoid obsolescence
A manually maintained sitemap becomes outdated as soon as the publication rate exceeds a few pages per month. Automation via the CMS or a deployment pipeline is the current best practice for sites that publish regularly.
On WordPress, plugins like Yoast SEO or Rank Math generate the XML sitemap with each publication or content update. The file then reflects the actual state of the site without human intervention.
Specialized sitemaps for rich content
Google allows declaring multiple types of content in sitemaps, beyond traditional HTML pages. Media-rich sites benefit from segmenting their files.
- Image sitemap: useful for portfolios, e-commerce sites, or image banks. Each image associated with a URL can be declared with its title and caption
- Video sitemap: relevant for sites hosting native video content. Metadata (duration, description, thumbnail) helps Google display rich snippets
- News sitemap: reserved for publishers listed in Google News, with a validity window of a few days after publication
A sitemap segmented by content type facilitates technical diagnosis. If video indexing poses a problem, the dedicated file allows isolating the cause without going through all the URLs of the site.
Submit and monitor the sitemap in Google Search Console
Declaring the sitemap in the robots.txt file via the directive Sitemap: https://example.com/sitemap.xml remains the reference passive method. All compliant crawlers automatically detect it during crawling.
Active submission via Google Search Console provides an additional level of information. The interface displays the number of detected URLs, the number of indexed URLs, and any errors reading the file. The gap between submitted URLs and indexed URLs serves as a direct indicator of the quality of the sitemap.
Reading the gap between submitted and indexed URLs
A small gap means that the file contains only URLs relevant in Google’s eyes. A large gap (the majority of submitted URLs are not indexed) points to a quality issue with the content, duplication, or conflicting directives.
This gap cannot be corrected by resubmitting the sitemap. It is necessary to analyze the excluded pages, understand the reason for rejection (thin content, noindex, canonical pointing to another URL), and clean the file accordingly.
The sitemap does not mechanically speed up indexing. It structures the dialogue between a site and crawlers, provided it only contains URLs that deserve to be explored. The true measure of its effectiveness is reflected in Google Search Console, in the report between what is submitted and what Google retains.