Free
Google Business Profile Audit
Audit your Google Maps listing: reviews, photos, hours and category, and see how you compare with competitors nearby.
Robots.txt & Sitemap Validator
Check that Google and AI crawlers can reach your pages, test any URL against your robots.txt, and validate your XML sitemap.
Checking…
Test a URL against robots.txt
Search & AI crawlers
Can each crawler read your homepage? Search/answer bots decide whether you can appear in Google, Bing, ChatGPT, Claude and Perplexity answers. Training bots only collect content for AI models.
Sitemaps
Sitemap URLs spot-check
A sample from your sitemap, checked the way Googlebot would see them.
Indexing Review
Pages not showing up on Google?
We find out exactly why pages aren't indexed, using your Search Console data, and fix robots.txt, sitemaps, canonicals and internal links so Google (and AI search) can find everything that matters.
- Search Console coverage and indexing audit
- Clean robots.txt and sitemaps that update themselves
- AI search visibility: ChatGPT, Gemini, Perplexity
What We Check
What this tool checks
Uses Google's own robots.txt rules (most specific crawler group, longest matching rule wins), so the result matches what Googlebot does.
robots.txt
Reachability, syntax, accidental "Disallow: /", blocked CSS/JS, and unsupported rules like Noindex.
AI crawlers
Whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended can read your site.
XML sitemaps
Valid XML, sitemap indexes, URL counts, wrong domains or http://, bad dates and duplicates.
URL spot-check
A sample of sitemap URLs checked for redirects, errors, noindex and robots.txt blocks.
More Free Tools
Try our other free tools
Free
SEO Checker
Score your page on 20+ SEO and local search factors, with plain-language fixes.
Free
Domain & SSL Expiry Checker
See when your domain and SSL certificate expire, plus who hosts your site and email.
Free
Website Authority Checker
Check your website's authority score and how it compares, with DA, PA and Spam Score where available.
Free
Meta Tag & SERP Preview
See how your page looks in Google and on WhatsApp/Facebook, edit the title and description live, and copy the tags.
Free
Email Security Checker
Check SPF, DKIM and DMARC: the settings that decide if your emails reach the inbox or spam, and stop others faking your address.
Free
Redirect Chain Checker
Follow every redirect for http, https, www and non-www, and catch loops, long chains and temporary redirects.
Free
UTM Link & QR Builder
Build trackable campaign links for ads, WhatsApp, Google Business Profile and flyers, with a QR code to print.
Free
Security Header Checker
Grade your website's HTTPS and browser security headers from A+ to F, with copy-paste fixes.
FAQs
Robots.txt & sitemaps: questions
Prefer to talk it through? Our team in Thane West is happy to help.
What does robots.txt do?
robots.txt is a small text file at yourdomain.com/robots.txt that tells search engines and other crawlers which parts of your site they may visit. One wrong line, such as "Disallow: /", can stop Google crawling your whole website.
Does robots.txt stop a page appearing in Google?
Not reliably. Blocking a page in robots.txt stops Google reading it, but the address can still appear in results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.
Should I block AI crawlers like GPTBot or ClaudeBot?
It depends on your goals. Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) collect content for AI models. Search and answer bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) fetch pages to cite them in AI answers. Most local businesses want to stay visible in AI answers, so they should at least allow the search bots.
What is an XML sitemap and do I need one?
An XML sitemap lists the pages you want search engines to find, with optional last-updated dates. Google can find pages without one, but a clean sitemap helps new and updated pages get discovered faster, especially on new sites or sites with few links.
Why are some of my sitemap URLs flagged?
Google expects sitemaps to list only final, indexable pages: addresses that load directly (status 200), are not blocked by robots.txt and have no noindex tag. Redirects, broken pages and noindex pages in a sitemap waste crawl time and send mixed signals.