What Is an XML Sitemap and Why Does Your Website Need One?
Most website owners know they have an XML sitemap because someone told them they needed one. An SEO agency set it up during the initial build. A plugin generated it automatically. Someone mentioned it in a technical audit. But ask what the sitemap actually does, why its configuration matters, or what a poorly maintained sitemap costs in rankings — and the answers become much less certain.
That knowledge gap matters, because an XML sitemap is not simply a box to tick. Configured correctly, it is one of the most effective tools available for directing Google’s attention to your most important pages. Configured incorrectly, it actively misleads Google — pointing crawlers towards pages you do not want indexed, including URLs that return errors, or failing to communicate your site’s most important content altogether.
This guide explains exactly what an XML sitemap is, how Google uses it, what correct configuration looks like, and what the most common sitemap errors are costing the sites that have them.
| Sitemaps in technical SEO audits: XML sitemap review is one of the first checks conducted in a professional technical SEO audit. Our guide on what happens during an SEO audit explains where sitemap analysis fits into the full investigation process. |
What Is an XML Sitemap?
An XML sitemap is a structured file that lists the URLs on your website that you want search engines to discover and index. It is written in XML (Extensible Markup Language), a plain-text format that search engines can read and process efficiently.
Think of it as a map of your website’s most important pages, handed directly to Google. Rather than relying entirely on Google’s crawlers to find every page by following links, you provide a clear, organised list of the URLs you consider most significant — along with some optional additional information about each one.
A basic XML sitemap looks like this:
| <?xml version=”1.0″ encoding=”UTF-8″?> <urlset xmlns=”http://www.sitemaps.org/schemas/sitemap/0.9″> <url> <loc>https://www.example.com/</loc> <lastmod>2026-01-15</lastmod> <changefreq>weekly</changefreq> <priority>1.0</priority> </url> <url> <loc>https://www.example.com/services/</loc> <lastmod>2025-11-20</lastmod> <changefreq>monthly</changefreq> <priority>0.8</priority> </url> </urlset> |
The sitemap itself is typically a file saved at the root of your website (yoursite.com/sitemap.xml) and submitted to Google Search Console. For larger sites, it may be broken into multiple sitemap files, all referenced by a sitemap index file.
How Google Uses Your XML Sitemap
Understanding how Google actually processes a sitemap corrects one of the most common misconceptions about them: a sitemap is not an instruction. It is a request.
Submitting a URL in your sitemap does not guarantee Google will index it. Google will still evaluate every URL it discovers for quality, relevance, and whether it deserves a place in the index. What the sitemap does is ensure Google knows the URL exists and knows you consider it important enough to specifically list.
Discovery
Google discovers new pages primarily by following links. But link discovery is not always complete or efficient. A page buried deep in a site’s architecture, a new page with no internal links yet, or a page on a large site that has not been recently crawled may take weeks or months to be found through link-following alone. A sitemap accelerates discovery by giving Googlebot a direct list of URLs to check, independent of the site’s internal link structure.
Crawl Budget Signalling
For large websites, crawl budget — the number of pages Googlebot will crawl in a given period — is a meaningful constraint. Google uses the sitemap as one of the signals for how to allocate that budget. Pages listed in the sitemap are implicitly flagged as more important than pages that are not. A well-configured sitemap helps direct Googlebot’s attention towards your commercially important pages and away from low-value URLs.
Freshness Signals
The lastmod tag in a sitemap entry tells Google when a page was last modified. When Googlebot sees a recent lastmod date on a URL it has already crawled, it prioritises recrawling that page to check whether the content has changed. This means sitemap lastmod dates can directly affect how quickly your content updates are discovered and reflected in search results.
Critically, lastmod dates must be accurate. If a site systematically sets every URL’s lastmod to today’s date regardless of whether the content has actually changed, Google quickly learns to ignore those signals entirely. Inaccurate lastmod dates undermine the trust Google places in the sitemap as a whole.
What Sitemaps Do Not Do
It is equally important to understand what a sitemap cannot do:
- It cannot force Google to index a URL. Quality and relevance still determine whether a page enters the index.
- It cannot override Google’s own judgement about canonical URLs. If Google has determined a different URL is the canonical version of a page, listing the non-canonical in the sitemap will not change that.
- It cannot prevent Google from crawling URLs not in the sitemap. Google can still discover and index pages through links even if they are not listed.
- It cannot substitute for good internal linking. A sitemap helps with discovery but internal links pass link equity, which a sitemap does not.
Why Your Website Needs an XML Sitemap
Not every website needs a sitemap equally. A five-page brochure site with a clean structure and strong internal linking will not gain dramatically from sitemap submission. But for the vast majority of websites actively managed for organic search, a correctly configured sitemap is a meaningful technical SEO advantage.
1. Large or Complex Websites
Google has a finite crawl budget for every site. On large sites with hundreds or thousands of pages, Googlebot cannot efficiently crawl every page in every crawl visit. Without a sitemap, Googlebot must rely entirely on following links from page to page, and pages that are poorly linked — or that sit deep in the site architecture — may be crawled infrequently or not at all. A sitemap ensures Google knows about every important page, regardless of its depth in the site structure.
2. New Websites with Few Inbound Links
A brand new website has no external links pointing to it and potentially very few internal links. Googlebot discovers sites primarily by following links from other sites. Without inbound links, a new site may not be crawled at all for weeks. Submitting a sitemap to Google Search Console accelerates the initial discovery process significantly.
3. Sites with Frequently Updated Content
News sites, blogs, e-commerce stores with regularly changing inventory, and any site that publishes new content frequently benefit from sitemap-driven freshness signalling. The lastmod tag communicates to Google that content has been updated and should be recrawled, helping time-sensitive content appear in search results promptly after publication or update.
4. Sites with Rich Media Content
Google offers specific sitemap extensions for images and videos that allow you to provide additional metadata about media content. An image sitemap can help Google index images that might not be easily discoverable through crawling alone. A video sitemap provides structured information about video content that supports video-specific search features.
5. Isolated Pages Without Strong Internal Links
On many websites, certain important pages are not well-connected to the rest of the site’s internal linking structure. Landing pages created for specific campaigns, new service pages that have not yet been linked from the main navigation, or older content that has lost its internal link context may be difficult for Googlebot to find through link-following alone. Listing these pages in the sitemap ensures they are discovered and crawled on schedule.
What an XML Sitemap Contains: The Elements Explained
| Element | Required? | What It Contains | SEO Implication |
|---|---|---|---|
| <loc> | Yes | The full absolute URL of the page (must include https://) | The primary content — every URL Google should know about |
| <lastmod> | No | Date the page was last significantly modified (YYYY-MM-DD format) | Inaccurate dates reduce Google’s trust in the sitemap’s freshness signals |
| <changefreq> | No | How often the page typically changes (always, hourly, daily, weekly, monthly, yearly, never) | Treated as a hint only. Google does not reliably follow changefreq values |
| <priority> | No | Relative importance of this URL vs others on your site (0.1 to 1.0) | Treated as a hint only. Google largely ignores priority values in practice |
| On priority and changefreq: Google’s own documentation acknowledges that changefreq and priority values are treated as hints that Google may or may not follow. Over-relying on these fields to influence crawl behaviour is largely ineffective. The most important elements are accurate loc values and genuine lastmod dates where content has actually changed. |
Types of Sitemaps
XML Sitemap (Standard)
The standard XML sitemap lists webpage URLs. This is the primary sitemap type for most websites and the one submitted to Google Search Console. It covers your homepage, service and product pages, blog posts, category pages, and any other content you want Google to index.
Sitemap Index File
A sitemap index file is a sitemap of sitemaps. When a website has more URLs than can fit in a single sitemap file (the limit is 50,000 URLs or 50MB uncompressed), multiple sitemap files are created and all referenced from a single index file. The sitemap index is what gets submitted to Search Console for large sites.
| <?xml version=”1.0″ encoding=”UTF-8″?> <sitemapindex xmlns=”http://www.sitemaps.org/schemas/sitemap/0.9″> <sitemap> <loc>https://www.example.com/sitemap-pages.xml</loc> <lastmod>2026-03-01</lastmod> </sitemap> <sitemap> <loc>https://www.example.com/sitemap-products.xml</loc> <lastmod>2026-03-01</lastmod> </sitemap> </sitemapindex> |
Image Sitemap
An image sitemap provides Google with information about images on your site that might not be easily discoverable through standard crawling. This is particularly valuable for e-commerce sites where product images are a key search asset, and for photography, design, or creative portfolios where image search visibility is commercially important.
Video Sitemap
A video sitemap provides structured information about video content including title, description, thumbnail URL, duration, and publication date. Sites with substantial video content — media publishers, course platforms, tutorial sites — benefit from video sitemaps for video-specific search feature eligibility.
News Sitemap
A news sitemap is a specialised format for content publishers eligible for Google News. It can only include articles published within the past 48 hours and follows specific formatting requirements. Only sites approved for Google News should use this sitemap type.
HTML Sitemap
An HTML sitemap is a human-readable page on the website that lists links to key pages. This is distinct from the XML sitemap — it is not submitted to search engines as a file but serves as a navigational aid for users and can improve internal linking. Both can coexist on the same site and serve complementary purposes.
What a Good Sitemap Looks Like vs a Problematic One
The difference between a sitemap that helps your rankings and one that actively undermines them is almost entirely in what URLs are included and whether those URLs are valid, indexable, and genuinely representative of your site’s most important content.
| Problematic sitemap | Correctly configured sitemap |
|---|---|
| Contains URLs that return 404 errors | Contains only URLs returning 200 status codes |
| Contains URLs that redirect to other pages | All URLs resolve directly to their final destination |
| Contains noindexed pages | All URLs are indexable (no noindex directive) |
| Contains thin or duplicate content pages | Contains only pages with genuine content value |
| Contains filter parameter URLs (e.g., ?sort=price) | Excludes URL parameter variants and faceted navigation URLs |
| lastmod dates set to today regardless of content changes | lastmod dates accurately reflect when content was last genuinely updated |
| Not submitted to Google Search Console | Submitted and verified in Search Console |
| URL count far exceeds actual valuable page count | URL count reflects actual indexable page inventory |
| Not referenced in robots.txt | Sitemap URL referenced in robots.txt |
| Never updated as site grows or changes | Automatically updated when new content is published |
The most damaging sitemap errors are almost always errors of inclusion rather than omission. Including URLs in your sitemap that are noindexed, redirecting, or returning errors tells Google there is a disconnect between what your sitemap claims and what your site actually contains. This reduces Google’s confidence in the sitemap as a reliable guide, which in turn reduces its effectiveness as a crawl direction tool.
Common XML Sitemap Errors and What They Cost You
These are the sitemap errors that appear most frequently in technical SEO audits, along with the specific ranking consequences of each.
| Sitemap Error | Severity | What to Check and Fix |
|---|---|---|
| Sitemap not submitted to Search Console | Critical | Google may not discover or regularly crawl the sitemap. Submit via Search Console > Indexing > Sitemaps |
| Sitemap contains URLs returning 404 errors | Critical | Signals to Google that the sitemap is not maintained. Remove all 404 URLs or redirect them and update the sitemap |
| Sitemap contains noindexed pages | High | Contradictory signals: sitemap says index, noindex tag says do not. Remove noindexed URLs from the sitemap immediately |
| Sitemap contains redirect URLs | High | Google must follow the redirect to find the actual canonical URL. Replace all redirecting URLs with their final destination URL |
| Filter parameter URLs in sitemap (e-commerce) | Critical | Massive crawl budget waste and thin content indexation at scale. Exclude all parameter URLs from sitemap generation |
| lastmod dates not updated when content changes | Medium | Google cannot use freshness signals to prioritise recrawl of updated content. Ensure CMS updates lastmod on content changes |
| Sitemap not referenced in robots.txt | Medium | Google may not discover the sitemap without a direct Search Console submission. Add Sitemap: directive to robots.txt |
| Sitemap file exceeds 50,000 URL limit | High | Google cannot fully process oversized sitemaps. Split into multiple files referenced by a sitemap index |
| Sitemap URL format uses relative paths | Critical | XML sitemaps require absolute URLs including protocol. Replace all relative paths with full absolute URLs |
| Sitemap includes low-value archive and tag pages | Medium | Dilutes the quality signal of the sitemap. Include only pages with genuine content value and ranking intent |
| Sitemap URL in robots.txt uses incorrect path | High | Google cannot find the sitemap from robots.txt. Verify the exact sitemap URL matches what is accessible on the server |
| Multiple sitemap plugins creating conflicting files | High | Duplicate or conflicting sitemaps create confusion about canonical URL set. Consolidate to a single sitemap source |
XML Sitemaps by Platform
How sitemaps are generated, configured, and updated varies significantly by CMS. Understanding the platform-specific implementation is important for correctly auditing and maintaining sitemap health.
WordPress
WordPress does not generate XML sitemaps natively (it added a basic sitemap in version 5.5 but with very limited configuration options). Most WordPress sites rely on SEO plugins for sitemap generation:
- Yoast SEO. Generates a sitemap index with sub-sitemaps per content type (posts, pages, categories, tags, authors). Configuration in SEO > General > Features > XML Sitemaps. Key settings include which post types and taxonomies to include and whether to exclude specific post types. The sitemap is available at yoursite.com/sitemap_index.xml.
- Rank Math. Generates sitemaps via Rank Math > Sitemap Settings. Provides more granular control over which post types and taxonomies are included. Sitemap available at yoursite.com/sitemap_index.xml.
- Common WordPress sitemap problems. Multiple plugins each generating sitemaps creating conflicts. Tag and author archive pages included in sitemaps when they should be excluded. Sitemap including posts set to noindex. Caching preventing sitemap from reflecting recent changes.
Full details on WordPress-specific sitemap audit are covered in our WordPress SEO audit guide.
Shopify
Shopify generates an XML sitemap automatically at yourstore.com/sitemap.xml with no configuration required. The sitemap includes sub-sitemaps for products, collections, pages, and blogs. Key considerations for Shopify sitemaps:
- The sitemap is automatically generated and updated by Shopify — it cannot be directly edited
- Filter parameter URLs generated by collection filtering apps may or may not appear in the sitemap depending on the app’s configuration
- Hidden products are excluded from the sitemap automatically
- The sitemap does not include the cart, account, or checkout pages by default
- Shopify’s sitemap.xml is the entry point that links to the sub-sitemaps — submit this to Search Console, not the individual sub-sitemaps
Full details on Shopify sitemap management are covered in our Shopify SEO audit guide.
Magento
Magento 2 includes a built-in sitemap generator configurable from Stores > Configuration > Catalog > XML Sitemap. Key configuration areas:
- Generation settings for products, categories, and CMS pages (frequency and priority per content type)
- Maximum file size and URL count per sitemap file (Magento automatically splits files when limits are exceeded)
- Whether to add the sitemap to robots.txt automatically (recommended to enable)
- Scheduled generation via cron job — verify the generation schedule is running correctly
- The sitemap must be generated manually or via cron after configuration — it does not generate automatically on first configuration
Full details on Magento sitemap audit are covered in our Magento SEO audit guide.
Custom and Headless Sites
Sites with custom-built frontends or headless CMS implementations must generate sitemaps programmatically. Key requirements:
- The sitemap generation process must update automatically when new content is published or existing content is modified
- The lastmod date must be sourced from the actual content modification date in the database, not from the file system or a static value
- The sitemap must exclude any URLs that are dynamically generated, parameter-based, or marked as noindex in the page’s meta tags
- For large sites, the sitemap generation process should be incremental rather than regenerating the full sitemap on every content change
How to Submit Your Sitemap to Google Search Console
Submitting your sitemap directly to Google Search Console ensures Google knows where to find it and allows you to monitor its processing status, including any errors Google identifies.
The submission process:
- Open Google Search Console and select the relevant property
- In the left-hand navigation, go to Indexing > Sitemaps
- In the ‘Add a new sitemap’ field, enter the path to your sitemap (e.g., sitemap.xml or sitemap_index.xml)
- Click Submit
- Google will process the sitemap and display the status, the number of URLs discovered, and any errors found
After submission, monitor the Sitemaps report regularly. The report shows how many URLs Google has discovered from the sitemap versus how many it has indexed. A significant gap between discovered and indexed URLs warrants investigation — it typically indicates either content quality issues, crawl budget constraints, or duplicate content preventing indexation. Our guide on indexing issues covers the full range of reasons pages fail to get indexed after submission.
| Also add to robots.txt: In addition to Search Console submission, reference your sitemap in your robots.txt file. Add the following line anywhere in the robots.txt file: Sitemap: https://www.yoursite.com/sitemap.xml This allows any search engine that reads robots.txt to discover your sitemap automatically, without relying solely on Search Console submission. |
How to Audit Your XML Sitemap
A sitemap audit has two objectives: confirming the sitemap contains only valid, indexable URLs, and verifying that all important pages are included. Here is the step-by-step process.
Step 1: Find and Access Your Sitemap
Most sitemaps are located at yoursite.com/sitemap.xml or yoursite.com/sitemap_index.xml. If these locations return a 404, check your SEO plugin settings or robots.txt file for the sitemap URL. The sitemap is also referenced in the Sitemaps report in Google Search Console.
Step 2: Check Search Console for Reported Errors
Open the Sitemaps report in Google Search Console. If Search Console has found errors in your sitemap, they will be displayed here. Common errors reported include URLs returning 4xx errors, URLs blocked by robots.txt, and sitemap formatting issues.
Step 3: Crawl the Sitemap with Screaming Frog
In Screaming Frog, go to Mode > List and enter your sitemap URL, or go to File > Upload and enter the sitemap URL directly. Screaming Frog will crawl every URL in the sitemap and return the response code, redirect chain, canonical tag, and noindex status for each one. This reveals:
- URLs returning 404 or other error status codes
- URLs that redirect (301 or 302) rather than resolving directly
- URLs with noindex meta tags or X-Robots-Tag headers
- URLs with canonical tags pointing to different URLs
Step 4: Cross-Reference Against Indexed Pages
In Google Search Console, go to Indexing > Pages and check how many pages are currently indexed. Compare this against the URL count in your sitemap. If the sitemap contains significantly more URLs than are indexed, identify which URL types are failing to be indexed and investigate the cause. If important pages are indexed but not in the sitemap, add them.
Step 5: Check for Missing Important Pages
Review the sitemap manually or by exporting the URL list and comparing against your site’s known page inventory. Common pages that are missing from sitemaps: new pages published without triggering a sitemap regeneration, pages on subdomains not covered by the main sitemap, and pages accessible via unusual URL paths not picked up by the sitemap generator.
Step 6: Verify robots.txt Reference
Access yoursite.com/robots.txt and check whether the sitemap URL is referenced with a Sitemap: directive. Verify that the URL in the directive exactly matches the accessible sitemap URL (correct protocol, www or non-www, exact file name).
XML Sitemap vs robots.txt: Understanding the Relationship
Sitemaps and robots.txt serve complementary but distinct purposes. Understanding how they interact prevents one of the most common and damaging configuration errors in technical SEO.
| Factor | XML Sitemap | robots.txt |
|---|---|---|
| Primary purpose | Tell search engines which pages you want crawled and indexed | Tell search engines which pages should not be crawled |
| Nature | Invitation — ‘please crawl these pages’ | Restriction — ‘do not crawl these paths’ |
| Effect on indexation | Does not guarantee indexation, only aids discovery | Disallowing a URL does not remove it from the index if already indexed |
| Format | XML file at yoursite.com/sitemap.xml | Plain text file at yoursite.com/robots.txt |
| Submitted to Search Console? | Yes — directly submitted for monitoring | Not submitted — automatically fetched by search engines |
| Can conflict? | Yes — including a URL in both sitemap and robots.txt disallow creates contradictory signals | Yes — blocking a sitemap URL in robots.txt prevents Google reading the sitemap |
| The most dangerous conflict: Including a URL in your XML sitemap while simultaneously blocking it with robots.txt is one of the most confusing signals you can send Google. The robots.txt disallow prevents Googlebot from crawling the URL, so it cannot verify what the page contains. The sitemap suggests the page is important and should be indexed. Google will typically not index the page (because it cannot crawl it) but may show it in search results with no description. More critically, Googlebot may not be able to read the sitemap itself if the sitemap file’s path is blocked in robots.txt. |
The correct approach is clear: robots.txt controls crawl access. The sitemap controls crawl prioritisation. Pages you want indexed belong in the sitemap and must not be blocked in robots.txt. Pages you want to prevent from being crawled belong in robots.txt and must not appear in the sitemap. Our guide on crawl errors explains the full range of crawl-related technical issues in detail.
Frequently Asked Questions
How often does Google check my sitemap?
Google re-fetches submitted sitemaps periodically, typically every few days for active sites. The exact frequency depends on the site’s crawl frequency, which is influenced by domain authority, content freshness, and historical crawl behaviour. You can request a re-fetch by resubmitting the sitemap in Google Search Console, which is particularly useful after a large content update or site migration.
What is the maximum size of an XML sitemap?
A single XML sitemap file can contain a maximum of 50,000 URLs and must not exceed 50MB when uncompressed. For sites exceeding these limits, multiple sitemap files should be created and referenced from a sitemap index file. The sitemap index file itself has no size limit in terms of the number of sitemaps it can reference, but each individual sitemap file must remain within the 50,000 URL and 50MB limits.
Should I include every page on my site in the sitemap?
No. The sitemap should include only pages you want Google to index and that have genuine content value. Pages that should typically be excluded include: pages with noindex tags, redirect pages, error pages, admin or login pages, duplicate or near-duplicate content pages, thin content pages, URL parameter variants, paginated pages beyond the first page (in most cases), and pages blocked by robots.txt. A smaller, higher-quality sitemap is more effective than a large sitemap diluted with low-value URLs.
Do I need a sitemap if my site is well-linked internally?
Internal links alone are sufficient for Google to discover all pages on a small, well-structured site. However, even for well-linked sites, a sitemap adds value by providing lastmod freshness signals, giving you a monitoring tool via Search Console, and ensuring new pages are discovered promptly without relying on crawl queue timing. The cost of maintaining a sitemap is negligible for most CMS platforms, making it worth having regardless of site size.
Why does my sitemap show more URLs than Google has indexed?
This is one of the most useful diagnostic signals in Search Console. A large gap between sitemap URLs and indexed URLs typically indicates one of four things: content quality issues preventing Google from considering the pages worth indexing, crawl budget constraints causing Googlebot to skip some pages in the sitemap, duplicate content causing Google to consolidate multiple URLs into a single indexed version, or technical issues preventing Google from accessing the pages despite their presence in the sitemap. Our guide on indexing issues covers the full diagnostic process for each of these scenarios.
How do I know if my sitemap has errors?
Google Search Console’s Sitemaps report is the primary source of sitemap error information. It shows the number of URLs discovered, the number indexed, and any specific errors Google has encountered. For more detailed analysis, crawling the sitemap URLs directly with Screaming Frog reveals response codes, redirect chains, canonical tags, and noindex directives across every URL in the sitemap. Both tools together give a complete picture of sitemap health. Sitemap audit is a standard component of every technical SEO audit we conduct at Web Pulse SEO.
Can I have multiple sitemaps for different content types?
Yes, and for large sites this is the recommended approach. Separate sitemaps for products, blog posts, category pages, and CMS pages make it easier to monitor indexation rates per content type and to identify which content types have the most indexation issues. All individual sitemaps are referenced from a single sitemap index file, which is what gets submitted to Search Console.
Is Your Sitemap Helping or Hindering Your Rankings?
A sitemap that contains 404 errors, noindexed pages, redirect URLs, or filter parameter variants is not a neutral asset — it is actively sending Google contradictory signals about your site’s content and reliability. For most sites that have been live for any length of time, the sitemap is worth auditing even if it was correctly configured at launch, because sites change and sitemaps frequently fail to keep pace.
XML sitemap review is a standard component of every technical SEO audit conducted by Web Pulse SEO. As part of the audit process, we verify every URL in the sitemap for response codes, canonical consistency, noindex conflicts, and robots.txt contradictions — and we identify any important pages that are missing from the sitemap entirely.
| Get started: Explore our technical SEO audit for a comprehensive investigation including full sitemap health review. Book a free consultation and we will assess your site’s sitemap and broader technical configuration. Browse our full audit services to find the right scope for your site. |
