Google’s Mobile Armageddon and Your Robots.txt File
If you work on websites, you’ve probably heard about Google’s “Mobile Armageddon”. Google is changing its search results to benefit sites that are mobile-friendly.
Google provided a mobile-friendly test that you can use to check whether your site will benefit from these new changes.
I used Google’s test for this site, and I was surprised that some of our main pages were being marked as “Not mobile friendly”, even though the pages clearly were mobile-friendly.
After digging deeper, I found out it was a simple fix and all I had to do was modify our robots.txt file to give Google access.
In this tutorial, I’ll show you how to check your site with Google and – if your site is responsive – how to resolve any issues with the robots.txt.
Background on the mobile armageddon
The robots.txt tells search engines what pages they shouldn’t index.
As more people have started using mobile devices, Google’s been modifying their search algorithms to make it easier for mobile devices to find mobile friendly sites.
In February, Google announced that on April 21st, they would rely more on their “mobile-friendliness” signal for ranking and that it would have a significant impact on search results.
The April 21st change is what some people on the web have termed as Google’s “Mobile Armageddon” since many sites that aren’t mobile friendly will likely have a significant loss in traffic over the coming months.
Checking whether Google thinks your site is mobile-friendly
By now, you’ve probably heard of “responsive design” and know that it helps to make your site mobile-friendly. A responsively designed web site adjusts to the width of the device viewing it.
If the site doesn’t adjust to the width of your mobile device and if you have to scroll horizontally to see the whole page, it’s not considered mobile-friendly and you’ll likely see a loss of ranking as a result.
If you navigate your site with a mobile device and it looks, navigates, and functions great, it’s mobile-friendly. However…
It doesn’t matter if you see it as mobile-friendly, but rather whether Google sees it as mobile-friendly.
If your site is mobile-friendly for you, go to https://www.google.com/webmasters/tools/mobile-friendly/ and double check that Google agrees with you. You should primarily check your home page and your main traffic generating pages. You might be surprised by the results, as I was.
Here’s a screenshot of a site that’s responsive, but is showing in Google as “Not mobile-friendly”.
Thankfully, Google’s tester is very informative and points out the exact issue for this particular site. It’s that the robots.txt file is blocking Google from accessing the resources it needs.
The robots.txt fix
Here’s the robots.txt file for the particular site.
It’s a Joomla site, which has much of the resources that Google needs stored in the “media” and “templates” folder. Also, there are some resources in “components”, “plugins”, and “modules”. So here are the lines that I’m going to remove:
Disallow: /components/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Update: Instead of removing the above, I can simply add the following rules underneath the rest of the code:
Allow: *.css
Allow: *.js
This will tell Google, and other search engines, to allow indexing of any .css and js files on the site.
You can find the exact resources that are blocked by Google by clicking on the “Show resources” link in Google’s tester. Compare the files there to your robots.txt to determine what needs unblocking.
Now Google agrees that we’re mobile-friendly
After saving your robots.txt file, go back to Google’s tester and run another test.
Awesome! This page is now mobile-friendly to Google as well.
Great tip. On my site though, google still says it’s not mobile friendly. When I check the site in PageSpeed Insights though, it reports user experience: 99/100. Huh???
Hi René,
What the site you’re working on?
The gakijken one that I found of yours still has the default Joomla robots.txt
Looking forward to helping.
That’s [url=http://www.avgm.nl]http://www.avgm.nl[/url]. I use the dynamic serving technique [url=https://developers.google.com/webmasters/mobile-sites/mobile-seo/configurations/dynamic-serving]https://developers.google.c…[/url]
First try removing the “Crawl-delay: 10” to test without it. I suspect that the mobile tester may be failing because of that line.
If that doesn’t help, you’d have to contact Google’s support for it, since it appears their mobile friendly test is not taking into account sites with dynamic serving.
Great write up! I’m very pleased to see Google supporting mobile friendly sites as a ranking method. Also nice to know that Google allows you to see if you pass their standards.
Thanks Josh!
I just have to tell you, I love you guys!! This one article helped me in so many ways!! THANK YOU!!
Glad to hear that, KarmicVisions! 🙂
help please,
I hate this google mobile friendly tester
first, it tells me that there are 4 resources that are blocked by my robots.txt file
well that’s interesting since I do not have a robots.txt file
so…
I put one in, and I verified that it was the new one using webmaster tools
the new file allows everything, blocks nothing
I used a 3rd party tester to test it
and it does not seem to be blocking the 4 resources the bobile friendly tester said it was
this tester is not seeing the new robots file
what do I do
I cannot learn how to make a site mobile friendly if this tester does not work
the url I was testing is
[url=http://www.rosemary.com]www.rosemary.com[/url]
thanks
Joomla! Dev team must include this changes to downloadable Joomla! package!
Thank you!
I think it’s already done with Joomla 3.4
Whats the link to the change so I can appeal for it NOT to be made?!
What do you mean ? I don’t understand. In Joomla 3.4, robots.txt have been updated in order to allow search engines to access Joomla’s media files. source : [url=https://www.ostraining.com/blog/joomla/joomla34/]https://www.ostraining.com/…[/url]
The change proposed by this blog post has NOT been applied to Joomla core, and neither should it be! The /media folder SHOULD be allowed, but the /components /plugins /modules should NEVER be.
see the current dev version of the robots.txt at:
[url=https://github.com/joomla/joomla-cms/blob/staging/robots.txt.dist]https://github.com/joomla/j…[/url]
@Phil
Actually, this blog post doesn’t propose a specific change. It shows an example site and how to check with Google which specific resources are blocked for their specific site. Maybe I didn’t make that clear though. I’ll read through the whole post later today and see where I can add clarification.
As to “/components /plugins /modules should NEVER be” is out-dated information. If an SEO rule is significantly hurting you, it shouldn’t be kept. Those specific rules have little to negligible benefit in 2015 with how advanced Google’s algorithms are, as well as the other major search engines who generally follow suit after Google.
Hi Nemo and Sébastien,
Great points!
“Disallow: /media/” was removed in Joomla 3.4 (it actually started 3.3.4). Also, “Disallow: /templates/” was removed at that time as well.
However, both those changes are for new installs of the Joomla core.
For Joomla 3 sites that were started prior to then, they would have to manually remove those entries from their robots.txt. When updating to Joomla 3.4, the robots.txt doesn’t get updated, so it needs to be manually done.
So for the vast major of current Joomla 3 installs, a manual examination of the robots.txt and a check with Google’s tester is required.
Hi Nick,
Thank you for these explanations. I thought robots.txt was automatically updated. So , i’ll need to check all my websites.
Cheers Sébastien
Rather than to fully Disallow some Joomla root folder you can be me more selective using the Allow directive for more details cf [url=https://compojoom.com/blog/entry/update-your-robots-txt#.VS6ZiBKl0tY.twitter]https://compojoom.com/blog/…[/url]
I actually think this is solving the problem the incorrect way. Google should never be allowed to freely index components/modules/plugins folders for risk of identifying “crap” (crap that should not be there to start with, but still the Joomla ecosphere is imperfect). The root issue is the Joomla extension history where an “addition” to Joomla would store all their assets in their own folder (not a bad thing really, encourages modularity) instead of the /images or /media folders (which are quite new, undocumented, and even not 100% used by Joomla). Making the proposed changes to the distributed robots.txt is the wrong approach. Its a quick fix that raises more issues than it solves. Also just because Google’s test doesnt pass – doesnt mean the site is not mobile friendly – it just means Google thinks its not, because Google cannot load assets!
Hmmm always pays to read the date of an article before replying – oh well..
This change has already been shot down and explained why it is a BAD THING to do in the following Joomla Core tracker – this article is missleading and doesnt cover anything apart from its own ends.
See the following
[url=https://github.com/joomla/joomla-cms/issues/6702]https://github.com/joomla/j…[/url]
[url=https://github.com/joomla/joomla-cms/pull/6839]https://github.com/joomla/j…[/url]
Actually, if you read through the discussion on github, you’ll realize that it doesn’t explain why it’s a bad thing, except for the fact that Joomla wouldn’t be forcing 3rd party developers to use the media folder by penalizing users with bad SEO. It’s a very poor reason to use an SEO rule, in my opinion.
Im not here to argue with you. Its wrong for all the right reasons. If Joomla made bad decisions like this then we would still be on Joomla 1.5 series. As developers we have to drag users kicking and screaming into 2015, when so many still run 1980s websites! Anyway the robots.txt is site specific so if you dont like the defaults, change it on YOUR sites, the defaults are sane and sensible for Joomla core. Robots is, and should be, specific for each and every site after it has been set live – the defaults are correct for a stock Joomla installation after install. Dont like it, change your site, not millions.
Generally, I agree 🙂 That’s why I submitted it just as a RFC. Either option would be sane and sensible 😛
That being said, this whole topic feels like users and developers will actually need to bring the Joomla core kicking and screaming into 2015 for SEO and overriding.
The current SEO rules in the robots.txt aren’t impactful in 2015 (they are OK, sane, and sensible, but not really impactful). They were back in the day, but search engines haven’t stayed still since then.
And the “you can only override media” and “everything should go in the media folder” seems archaic to a lot of us 🙂 I think the community knowledge and expertise has matured enough to make everything overrideable by now. Hopefully, Roberto, PLT, and the community will be able to bring that to fruition in J 3.5.
Thanks a lot, i face the same problem, and i use your tips now my site is
Awesome! This page is mobile-friendly.
Thanks again for nice tricks.
I’m glad it helped, Bulbul! Thanks for sharing
Instead of removing components, modules etc from robot.txt just add this
#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js
Hi Aaqib,
Great suggestion! After writing this post, a few weeks later I saw someone else use the above approach for a Drupal site. It’s simple and effective, and I don’t see any issues with it. I’ll go ahead and add it as an update to this post.
Thanks Nick, this would be great help for people having similar issue. I tried that solution on my Joomla website and it worked perfect.
I’ve been trying to work through this with some of my Joomla websites and still have issues with some.
I’ve tried:
• Commenting out certain directories
→ That did not work
• Adding the Allow options as described here
→ On one site in particular this does not work
Any suggestions…?
Hi Michael,
What’s the link to your site?
Hi Nick and thanks for the response.
The site in question is:
[url=http://www.divingtenerifescuba.com]http://www.divingtenerifesc…[/url]
Your robots.txt file is OK and it passes Google’s test.
Looks like Google’s just timing out on some of your images. The first time I tested, it timed out. The second time, everything worked. In both cases, Google said it was responsive and that it passed its test.
Since the Google test passes, I wouldn’t be concerned about it.
Hope this helps!
It does. I was kind of thinking the same myself so it’s nice to have that confirmed.
Hi Nick,
Thanks for the info, I’m still really struggling to fix this issue on my site. here’s my robot.txt file…
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Allow: *.css
Allow: *.js
Hi Martin,
What’s your site?
Thanks Nick it’s [url=http://www.cd-duplication-services.co.uk]www.cd-duplication-services…[/url]
Thanks
Strange indeed. Try eliminating the spaces between your disallows and allows.
Done this. Still not worked 🙁
[url=https://www.google.co.uk/webmasters/tools/mobile-friendly/?url=www.cd-duplication-services.co.uk]https://www.google.co.uk/we…[/url]
Fixed!! This fixed the issue straight away. see below
To fix that problem, add this rule to the TOP of your robots.txt file, which gives Google permission to index js and css files regardless of where it finds them:
#Googlebot
User-agent: Googlebot
Allow: *.css
Allow: *.js
Glad you got things sorted, Martin!
Hi Nick it’s www. cd-duplication-services .co.uk
Thanks