What Is an XML Sitemap?

What an XML Sitemap Is

An XML sitemap is a file that lists the URLs of a website. It serves as a roadmap for search engine crawlers like Googlebot, telling them which pages are available for indexing. By providing a sitemap, you can help search engines discover all the important pages on your site, including those that might not be easily found through standard crawling or internal linking alone.

The sitemap format uses XML (Extensible Markup Language), which makes it machine-readable. It is not intended for human visitors but for automated systems.

Here is a basic example of an XML sitemap for a small website:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>https://www.example.com/</loc>
      <lastmod>2023-10-26</lastmod>
      <changefreq>daily</changefreq>
      <priority>1.0</priority>
   </url>
   <url>
      <loc>https://www.example.com/about</loc>
      <lastmod>2023-10-25</lastmod>
      <changefreq>weekly</changefreq>
      <priority>0.8</priority>
   </url>
   <url>
      <loc>https://www.example.com/products/product-a</loc>
      <lastmod>2023-10-24</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.7</priority>
   </url>
</urlset>

This example shows a sitemap declaring three URLs for a website. Each URL entry includes <loc> (location) as a required element, and may include optional elements such as <lastmod> (last modified date), <changefreq> (change frequency), and <priority> (priority relative to other URLs on the site).

When You Need One (and When You Don't)

XML sitemaps are most beneficial for certain types of websites and less critical for others.

You generally need an XML sitemap if your site:

• Is large: Websites with thousands or millions of pages, such as large e-commerce sites or news archives, can benefit from sitemaps to ensure all content is discoverable.
• Has isolated pages: If some pages on your site are not well-linked internally, a sitemap can help search engines find them. This can occur with deep content categories or new pages not yet fully integrated into your site's navigation.
• Is new: New websites often have few backlinks, which search engines use to discover content. A sitemap can accelerate the discovery and indexing process.
• Contains rich media: Sitemaps can include specific extensions for images, videos, and news content, helping search engines understand and index these specialized media types.
• Uses dynamic content: Sites where content is generated dynamically or through user interaction may have URLs that are difficult for crawlers to find without explicit listing in a sitemap.

You might not strictly need an XML sitemap if your site:

• Is small: If your site has fewer than 500 pages and all pages are adequately linked internally, search engines can usually crawl and index your entire site without a sitemap.
• Has comprehensive internal linking: If every page on your site is reachable from every other page through a reasonable number of links, crawlers can typically discover all content.
• Is well-established with many backlinks: For sites that have existed for a long time and accumulated a strong backlink profile, search engines may already be familiar with most of your content.

Even in cases where a sitemap is not strictly necessary, providing one is generally considered a best practice for SEO. It offers an additional signal to search engines and costs little to implement.

Sitemap Structure and Required Fields

The basic structure of an XML sitemap involves a root <urlset> element containing one or more <url> child elements. Each <url> element represents a distinct URL on your website.

Within each <url> element, the only mandatory field is <loc>, which specifies the absolute URL of the page. This URL must be fully qualified, including the protocol (e.g., https://).

Optional fields include:

Field tag	Description	Example
<lastmod>	The date of last modification of the file. This helps search engines understand which content has been recently updated. Google uses this information to determine when to recrawl pages.	2023-10-26
<changefreq>	An indication of how frequently the page is likely to change. Valid values include always, hourly, daily, weekly, monthly, yearly, and never. Google generally treats this as a hint, not a directive.	daily
<priority>	A value between 0.0 and 1.0 that assigns a priority to a URL relative to other URLs on the same site. Higher values indicate more important pages. The default priority is 0.5. Google rarely uses this value for ranking.	0.8

While <priority> and <changefreq> are part of the sitemaps protocol, Google has indicated that it generally ignores these values for crawling and ranking decisions. The <lastmod> date is the most impactful optional field, as Google actively uses it to decide when a page might need re-crawling. Keeping <lastmod> accurate can help ensure that updated content is indexed faster.

Sitemap Size Limits and Index Files

Search engines impose limits on the size and number of URLs within a single XML sitemap file. For Google, a single sitemap file cannot contain more than 50,000 URLs and must not exceed 50MB in uncompressed size.

If your website exceeds these limits, you must split your sitemap into multiple smaller sitemap files. To manage these multiple files, you would create a "sitemap index file." A sitemap index file is another XML file that lists all your individual sitemap files.

Here is an example of a sitemap index file:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <sitemap>
      <loc>https://www.example.com/sitemap_products_1.xml</loc>
      <lastmod>2023-10-26</lastmod>
   </sitemap>
   <sitemap>
      <loc>https://www.example.com/sitemap_products_2.xml</loc>
      <lastmod>2023-10-26</lastmod>
   </sitemap>
   <sitemap>
      <loc>https://www.example.com/sitemap_blog.xml</loc>
      <lastmod>2023-10-25</lastmod>
   </sitemap>
</sitemapindex>

Each <sitemap> entry in the index file includes the <loc> of an individual sitemap and an optional <lastmod> date for that sitemap file. This structure allows search engines to read a single index file and then discover all associated sitemaps, collectively covering all of your site's URLs.

Submitting to Google Search Console

After creating your XML sitemap, the next step is to submit it to Google Search Console (GSC). This informs Google about the existence and location of your sitemap.

To submit your sitemap to GSC:

1. Verify your website: If you haven't already, add and verify your website property in Google Search Console.
2. Navigate to the Sitemaps report: In the left-hand navigation pane of GSC, click on "Sitemaps" under the "Indexing" section.
3. Add a new sitemap: In the "Add a new sitemap" section, enter the full URL to your sitemap file (e.g., sitemap.xml or sitemap_index.xml) relative to your domain. For example, if your sitemap is located at https://www.example.com/sitemap.xml, you would enter sitemap.xml.
4. Click "Submit": GSC will then attempt to process your sitemap.

Once submitted, GSC will display the status of your sitemap, showing when it was last read, how many URLs were discovered, and any potential errors it encountered. It typically takes some time for Google to process a submitted sitemap and for the changes to reflect in indexing.

You can also specify your sitemap's location in your robots.txt file using the Sitemap directive. This provides an additional discovery mechanism for search engines.

User-agent: *
Disallow: /wp-admin/
Sitemap: https://www.example.com/sitemap.xml

This line should be placed at the end of your robots.txt file.

Common Sitemap Mistakes

Several common mistakes can prevent a sitemap from being effective or even lead to indexing issues:

• Including non-canonical URLs: Only include the canonical version of each URL. If a page has duplicate content at different URLs, ensure only the preferred (canonical) URL is in the sitemap. Including non-canonical URLs can confuse search engines about which version to index.
• Including noindex URLs: Do not list URLs in your sitemap that are blocked from indexing by a noindex meta tag or X-Robots-Tag HTTP header. The sitemap suggests pages for indexing; conflicting signals can create uncertainty for crawlers.
• Including blocked URLs in robots.txt: Do not include URLs in your sitemap that are disallowed by your robots.txt file. If robots.txt prevents crawlers from accessing a page, listing it in the sitemap is contradictory and will not lead to indexing.
• Not using absolute URLs: All URLs listed in <loc> tags must be absolute, including the protocol (e.g., https://www.example.com/page.html). Relative URLs will be ignored.
• Having incorrect lastmod dates: While beneficial, lastmod dates should be accurate. Listing a date in the future, or a date that does not reflect a true modification, can mislead search engines about content freshness.
• Broken URLs: Ensure all URLs listed in your sitemap respond with a 200 OK status code. Including broken links (404s) can waste crawler resources and signal low quality.
• Improper XML formatting: Sitemaps must adhere strictly to XML formatting rules. Errors such as unclosed tags, incorrect character encoding, or invalid namespaces can cause the sitemap to be unreadable by search engines. Tools like the LemRank Sitemap Validator can help identify these issues.
• Forgetting to update: Sitemaps should be kept up-to-date as your website content changes. New pages should be added, and deleted pages should be removed. Forgetting to update your sitemap means search engines might miss new content or attempt to crawl non-existent pages.

Regularly validating and updating your sitemap, along with monitoring its status in Google Search Console, helps ensure it remains an effective tool for SEO.

Related

• How to Use the Sitemap Validator
• How to Use the robots.txt Tester