XML sitemap best practices: what to include and what to leave out

A sitemap is a promise. Each URL in it says to search engines: this page exists, it is the version I want indexed, and it is worth your crawl. When sitemaps are generated carelessly, that promise gets broken hundreds of times, and the file becomes noise. Following a handful of XML sitemap best practices keeps it a reliable signal, which matters most for new sites, large catalogues and pages that are not well linked internally.

This article is about what the file should contain. Submitting it and reading Search Console errors is a separate job worth its own checklist.

What a sitemap can and cannot do

An XML sitemap helps search engines discover URLs and understand which ones you consider important. It does not force indexing. Google still decides whether each page is useful enough to index, and it will still find pages through links whether they are in the sitemap or not.

Small sites with good internal linking can often manage without one, but a clean sitemap costs little and gives useful reporting in Google Search Console. For sites with thousands of products, frequent new articles or deep archives, it genuinely speeds up discovery.

Include only canonical, indexable URLs

The single most important rule. Every URL in the sitemap should meet all of these conditions:

• Returns a 200 status code, not a redirect or error
• Is not marked noindex
• Is not blocked by robots.txt
• Is the canonical version, meaning its canonical tag points to itself
• Uses your preferred protocol and hostname, for example https with or without www, exactly as the live site does
• Is a page you would be happy to see in search results

When a URL fails any of these, including it sends mixed signals. A sitemap listing http://example.in/about while the canonical is https://www.example.in/about asks Google to reconcile a contradiction you created.

What to leave out of your sitemap

Redirected URLs

After a redesign or URL change, old addresses often linger in sitemaps. List the destination instead. Redirecting URLs in a sitemap waste crawl requests and appear as issues in reports.

Noindex pages

Thank-you pages, internal search results, login and account pages, cart and checkout pages are commonly noindexed, correctly. Listing them in the sitemap contradicts that instruction.

Parameter and filter URLs

Sorted, filtered or tracked versions such as ?sort=price or ?utm_source=whatsapp are duplicates of the main page and should not appear.

Thin archive pages

Tag pages with one post, date archives, author archives on a single-author blog, and attachment pages that exist for each uploaded image usually add little. Many WordPress SEO plugins let you exclude these content types from the sitemap in a few clicks.

Paginated duplicates and test content

Staging pages, sample products left from the theme demo, and "Hello world" posts should be deleted or excluded, not advertised.

Keep lastmod accurate or leave it out

The lastmod field tells search engines when a page last changed meaningfully. Google has said it uses lastmod when it is consistently and verifiably accurate, and ignores it on sites where it is not.

Common ways lastmod goes wrong:

• Every URL showing today’s date because the sitemap is regenerated daily
• Dates updating when only a sidebar widget or footer year changed
• Dates that never change even after major content edits

Set lastmod from the date the main content was actually edited, such as the post’s updated timestamp or the product’s last price or description change. If your platform cannot do that reliably, omitting lastmod is better than publishing misleading values.

Two other fields, changefreq and priority, are ignored by Google, so there is no need to agonise over them.

Splitting large sitemaps sensibly

The sitemap protocol limits a single file to 50,000 URLs or 50MB uncompressed. Bigger sites must split into several sitemaps and list them in a sitemap index file. Even below the limit, splitting has benefits:

• By content type: separate files for pages, blog posts, products and categories
• By section: for a news portal, by year or month of publication
• By size: products in chunks of, say, 10,000 so no file grows unwieldy

Search Console reports indexing per submitted sitemap, so logical splits make problems easier to locate. If product pages show poor indexing but blog posts are fine, you know where to look.

Keeping the sitemap in sync with the site

A sitemap that is written once by hand and forgotten becomes inaccurate within months. Generate it dynamically from your CMS or database so it updates when pages are published, unpublished or redirected. Then add a few safeguards:

1. When a product goes permanently out of stock and its page is removed, make sure the URL drops out of the sitemap.
2. When a page is set to noindex, confirm the generator excludes it automatically.
3. After a migration, crawl every sitemap URL and check each returns 200 with a self-referencing canonical.
4. Reference the sitemap location in robots.txt with a Sitemap line, so any crawler can find it.
5. Re-crawl the sitemap quarterly with an SEO crawler’s list mode to catch drift.

A quick sitemap health check

Open your sitemap in a browser, usually at /sitemap.xml or /sitemap_index.xml, and answer these questions:

• Do all URLs use https and your chosen hostname?
• Can you spot any thank-you, cart, login or search result pages?
• Are there demo, test or placeholder pages?
• Do lastmod dates look plausible, or are they all identical?
• Does the number of URLs roughly match the number of real pages you have?

If the count is wildly higher than your real page count, something is generating junk URLs; if it is much lower, important sections are missing.

Cleaning up your own sitemap

Run the health check above, fix the generator settings rather than editing the file by hand, and re-check a week later. On a CMS this is often a matter of plugin settings; on a custom platform it may need a small code change. If your site is custom-built and the sitemap is out of date, our software development team can connect it to your content so it stays accurate automatically.

Frequently asked questions

Should every page on my website be in the sitemap?

No. Include only pages you want indexed. Utility pages such as carts, logins and thank-you pages should be left out.

Do I need a separate sitemap for images or videos?

Not necessarily. Image and video details can be added to your existing sitemap using the relevant extensions, or placed in separate sitemaps if that is easier to manage.

Is an HTML sitemap the same as an XML sitemap?

No. An HTML sitemap is a page of links for visitors, while an XML sitemap is a machine-readable file for search engines. A site can have either, both or neither.

Does having a sitemap improve rankings?

Not directly. It helps pages get discovered and crawled, which is a prerequisite for ranking, but ranking itself depends on content quality and relevance.