Crawling slows down site-wide
A 5xx on robots.txt can make Googlebot back off the whole site until the file responds again. No page breaks, nothing looks wrong.
Robots.txt monitoring
Danger
One line in robots.txt can hide your whole site
A staging config, a plugin update, a generator left running — and the one file that tells crawlers what they may read says something else this morning. The site still loads. Nothing on the page tells you.
Danger severity: you get the alert within the hour, not in tomorrow's digest.
What it may cost you
The file keeps returning 200 and the site keeps loading. That is exactly why this one goes unnoticed.
Rules you rely on go silent
Google reads the first 500 KB and ignores the rest. The rules that get cut are the ones appended last — usually the ones you added most recently.
Nobody remembers editing it
The file is edited by hand and by plugins far more often than anyone tracks. A Disallow: / from a staging config reaches production and stays there.
How an alert reaches you
Three steps, running every day, on something nobody looks at twice a year.
1
Every run
Baseline stored
The first crawl saves your robots.txt as it should be. That value is what every later run is compared against.
2
Next morning
Change detected
The file is fetched, parsed and diffed against the stored copy. Added and removed lines are highlighted, so you read the change instead of the file.
3
Within the hour
Alert in your inbox
Danger-level findings go out right after the crawl batch. Milder ones wait for the daily digest, so the urgent one stands out.
You decide what counts as broken
The stored value is your desired state, not a guess. When a later run differs you get both versions side by side — and if the change was intentional, one click adopts it as the new baseline.
Severity matches the damage
This one reaches you within the hour, because the damage starts before the next digest would arrive. Every test's severity can be overridden per site, so your inbox stays useful.
Set it once, then leave it
Pick the frequency and the user agent Wygard should send, switch on JS rendering if your pages need it, and the rest needs no attention.
What we check
Five checks guard the file’s availability, size, syntax, sitemap references and content. All of them are in Basic and on by default.
| Test | What it catches | Scope | Severity |
|---|---|---|---|
| Robots.txt — status code | The file stops returning 200 — a 404 after a deploy, a 5xx from an overloaded server. |
Web · Basic | Danger |
| Robots.txt — max file size | The file grows past Google’s 500 KB cap, so everything below the cutoff is ignored. | Web · Basic | Danger |
| Robots.txt — content changes | Any line added or removed since the last run, shown as a diff. | Web · Basic | Depends on the reason |
| Robots.txt — validity | Syntax a crawler can’t reliably read — unknown directives, two rules on one line, Crawl-delay under Googlebot. |
Web · Basic | Depends on the reason |
| Robots.txt — live sitemap included | A Sitemap: line pointing at a URL that no longer resolves. |
Web · Basic | Depends on the reason |
Questions about robots.txt monitoring
What the checks do, and what they don’t.
How often does Wygard check robots.txt
Daily by default. You can set weekly or monthly per test, and custom frequencies are available on the Advanced plan.
What happens when I edit robots.txt on purpose
You get the alert with the diff, and one click on Set found value as desired adopts the new file as the baseline. The next run is green.
Does this replace a crawler
No. Wygard watches a small set of resources continuously instead of crawling everything occasionally. Keep your crawler for discovery, use Wygard to know when something moved.
Which plan includes the robots.txt checks
All five are in Basic and on by default, one robots.txt per site.
When this broke for real
What a silent failure of this kind looks like from the inside, and how long it took to catch.
SEO fail stories
The robots.txt mistake that blocked Google in plain sight
The file read fine to a human and a popular tester called it allowed — but Google drops Crawl-delay, merges the User-agent lines into one group, and applies the Disallow: / below them to the whole site.
“This app has completely transformed how we manage our SEO. The real-time alerts ensure we never miss an issue. It's a game-changer for protecting our rankings!”
Why Wygard?
Trust, but verify. You build the site — Wygard makes sure your key SEO elements work as intended.
Your SEO protected
How quickly would you notice if your canonical tags changed? We catch it for you.
Time saved
Don't waste time on massive crawls and endless data. With 15 years of hands-on SEO experience, we know how to focus on what really matters.
Easy to Use
You don't have to be an SEO expert — your site still deserves expert monitoring. That's exactly what our tool delivers.
Single site or the whole portfolio? We can handle all today
Every online business deserves protection. No matter the size.