An XML sitemap is your direct blueprint for search engines: it tells Googlebot which URLs on your website actually matter and when they were last updated. Yet on most growing blogs, Shopify stores, and WordPress sites, sitemaps quietly accumulate 301 redirects, 404s, tag archives, and non-canonical URLs—teaching Googlebot to ignore them.
This guide covers the exact XML sitemap rules that maximize crawl trust, plus how to pair a clean sitemap with an automated URL indexing tool so every newly published page gets crawled within hours instead of weeks.
The Golden Rule: Only Indexable, Canonical HTTP 200 URLs
Every single <loc> entry inside your XML sitemap should pass four strict checks:
- Returns HTTP 200 OK directly (zero 301/302 redirects, zero 404/410s).
- Is the canonical version of the page (never list parameter duplicates or alternate URLs).
- Has no
noindexmeta tag orX-Robots-Tagheader. - Is not blocked by
robots.txt.
Not sure if your sitemap URLs are clean? Paste a sample into our Free URL Indexability Checker before submitting.
Technical Sitemap Limits & Structure
| Sitemap Element | Google Rule / Best Practice | Common Mistake |
|---|---|---|
| Max URLs per file | 50,000 URLs (or 50 MB uncompressed); split via sitemap_index.xml |
Dumping 100k+ URLs into one giant file |
<lastmod> |
Use accurate ISO-8601 dates updated only on meaningful content edits | Auto-stamping every URL with today's date on every build |
<priority> & <changefreq> |
Ignored by Google (harmless to keep for Bing) | Setting every URL to 1.0 thinking it boosts rankings |
robots.txt reference |
Always declare Sitemap: https://yourdomain.com/sitemap.xml |
Forgetting to update after moving from HTTP to HTTPS |
Verify Your Sitemap From the Command Line
curl -sI https://yourdomain.com/sitemap.xml | grep -iE "^HTTP/|content-type:"
Confirm it returns HTTP/2 200 with application/xml or text/xml—never an HTML error page or redirect loop.
Automate Sitemap Indexing (Publish & Forget)
Even a spotless XML sitemap relies on Google's passive crawl schedule. With MonkeyIndexer Automated Sitemap Indexing, you connect your sitemap.xml or nested Yoast/RankMath sitemap_index.xml once:
- MonkeyIndexer checks your sitemap automatically every 6 hours.
- Only brand-new URLs are submitted—previously seen pages are never double-charged.
- If your wallet ever runs low, newly detected URLs park safely as Pending Credit and resume automatically once topped up.
Frequently Asked Questions
Should I include blog tag and author archives in my XML sitemap?
Usually no. Unless your tag pages are curated landing pages with unique copy and search demand, excluding thin archives keeps your sitemap tight and prevents index bloat.
Does MonkeyIndexer work with Yoast, RankMath, and Shopify sitemaps?
Yes—both standard XML sitemaps and nested sitemap index files are parsed automatically.