Google’s Mobile Armageddon and Your Robots.txt File

If you work on websites, you’ve probably heard about Google’s “Mobile Armageddon”. Google is changing its search results to benefit sites that are mobile-friendly.

Google provided a mobile-friendly test that you can use to check whether your site will benefit from these new changes.

I used Google’s test for this site, and I was surprised that some of our main pages were being marked as “Not mobile friendly”, even though the pages clearly were mobile-friendly.

After digging deeper, I found out it was a simple fix and all I had to do was modify our robots.txt file to give Google access.

In this tutorial, I’ll show you how to check your site with Google and – if your site is responsive – how to resolve any issues with the robots.txt.

Background on the mobile armageddon

The robots.txt tells search engines what pages they shouldn’t index.

As more people have started using mobile devices, Google’s been modifying their search algorithms to make it easier for mobile devices to find mobile friendly sites.

In February, Google announced that on April 21st, they would rely more on their “mobile-friendliness” signal for ranking and that it would have a significant impact on search results.

The April 21st change is what some people on the web have termed as Google’s “Mobile Armageddon” since many sites that aren’t mobile friendly will likely have a significant loss in traffic over the coming months.

Checking whether Google thinks your site is mobile-friendly

By now, you’ve probably heard of “responsive design” and know that it helps to make your site mobile-friendly. A responsively designed web site adjusts to the width of the device viewing it.

If the site doesn’t adjust to the width of your mobile device and if you have to scroll horizontally to see the whole page, it’s not considered mobile-friendly and you’ll likely see a loss of ranking as a result.

If you navigate your site with a mobile device and it looks, navigates, and functions great, it’s mobile-friendly. However…

It doesn’t matter if you see it as mobile-friendly, but rather whether Google sees it as mobile-friendly.

If your site is mobile-friendly for you, go to https://www.google.com/webmasters/tools/mobile-friendly/ and double check that Google agrees with you. You should primarily check your home page and your main traffic generating pages. You might be surprised by the results, as I was.

Here’s a screenshot of a site that’s responsive, but is showing in Google as “Not mobile-friendly”.

Thankfully, Google’s tester is very informative and points out the exact issue for this particular site. It’s that the robots.txt file is blocking Google from accessing the resources it needs.

The robots.txt fix

Here’s the robots.txt file for the particular site.

It’s a Joomla site, which has much of the resources that Google needs stored in the “media” and “templates” folder. Also, there are some resources in “components”, “plugins”, and “modules”. So here are the lines that I’m going to remove:

{codecitation}Disallow: /components/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/{/codecitation}

Update: Instead of removing the above, I can simply add the following rules underneath the rest of the code:

{codecitation}Allow: *.css
Allow: *.js{/codecitation}

This will tell Google, and other search engines, to allow indexing of any .css and js files on the site.

You can find the exact resources that are blocked by Google by clicking on the “Show resources” link in Google’s tester. Compare the files there to your robots.txt to determine what needs unblocking.

Now Google agrees that we’re mobile-friendly

After saving your robots.txt file, go back to Google’s tester and run another test.

Awesome! This page is now mobile-friendly to Google as well.

Instructor

  • Nick is the Director of Support at OSTraining and you can find him in almost every area of the site, from answering support requests and account questions to creating tutorials and software.

0 0 votes
Blog Rating
Subscribe
Notify of
43 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
renekreijveld

Great tip. On my site though, google still says it’s not mobile friendly. When I check the site in PageSpeed Insights though, it reports user experience: 99/100. Huh???

renekreijveld

That’s [url=http://www.avgm.nl]http://www.avgm.nl[/url]. I use the dynamic serving technique [url=https://developers.google.com/webmasters/mobile-sites/mobile-seo/configurations/dynamic-serving]https://developers.google.c…[/url]

Josh Lewis

Great write up! I’m very pleased to see Google supporting mobile friendly sites as a ranking method. Also nice to know that Google allows you to see if you pass their standards.

KarmicVisions

I just have to tell you, I love you guys!! This one article helped me in so many ways!! THANK YOU!!

Dennis Oppelt

help please,

I hate this google mobile friendly tester

first, it tells me that there are 4 resources that are blocked by my robots.txt file

well that’s interesting since I do not have a robots.txt file

so…

I put one in, and I verified that it was the new one using webmaster tools

the new file allows everything, blocks nothing

I used a 3rd party tester to test it

and it does not seem to be blocking the 4 resources the bobile friendly tester said it was

this tester is not seeing the new robots file

what do I do

I cannot learn how to make a site mobile friendly if this tester does not work

the url I was testing is

[url=http://www.rosemary.com]www.rosemary.com[/url]

thanks

Nemo

Joomla! Dev team must include this changes to downloadable Joomla! package!

Thank you!

Sébastien Gicquel

I think it’s already done with Joomla 3.4

Phil Taylor, The Joomla Expert

Whats the link to the change so I can appeal for it NOT to be made?!

Sébastien Gicquel

What do you mean ? I don’t understand. In Joomla 3.4, robots.txt have been updated in order to allow search engines to access Joomla’s media files. source : [url=https://www.ostraining.com/blog/joomla/joomla34/]https://www.ostraining.com/…[/url]

Phil Taylor, The Joomla Expert

The change proposed by this blog post has NOT been applied to Joomla core, and neither should it be! The /media folder SHOULD be allowed, but the /components /plugins /modules should NEVER be.
see the current dev version of the robots.txt at:

[url=https://github.com/joomla/joomla-cms/blob/staging/robots.txt.dist]https://github.com/joomla/j…[/url]

Sébastien Gicquel

Hi Nick,

Thank you for these explanations. I thought robots.txt was automatically updated. So , i’ll need to check all my websites.

Thierry Tardif - Alpha Design

Rather than to fully Disallow some Joomla root folder you can be me more selective using the Allow directive for more details cf [url=https://compojoom.com/blog/entry/update-your-robots-txt#.VS6ZiBKl0tY.twitter]https://compojoom.com/blog/…[/url]

Phil Taylor, The Joomla Expert

I actually think this is solving the problem the incorrect way. Google should never be allowed to freely index components/modules/plugins folders for risk of identifying “crap” (crap that should not be there to start with, but still the Joomla ecosphere is imperfect). The root issue is the Joomla extension history where an “addition” to Joomla would store all their assets in their own folder (not a bad thing really, encourages modularity) instead of the /images or /media folders (which are quite new, undocumented, and even not 100% used by Joomla). Making the proposed changes to the distributed robots.txt is the wrong approach. Its a quick fix that raises more issues than it solves. Also just because Google’s test doesnt pass – doesnt mean the site is not mobile friendly – it just means Google thinks its not, because Google cannot load assets!

Phil Taylor, The Joomla Expert

Hmmm always pays to read the date of an article before replying – oh well..

Phil Taylor, The Joomla Expert

This change has already been shot down and explained why it is a BAD THING to do in the following Joomla Core tracker – this article is missleading and doesnt cover anything apart from its own ends.
See the following

[url=https://github.com/joomla/joomla-cms/issues/6702]https://github.com/joomla/j…[/url]

[url=https://github.com/joomla/joomla-cms/pull/6839]https://github.com/joomla/j…[/url]

Phil Taylor, The Joomla Expert

Im not here to argue with you. Its wrong for all the right reasons. If Joomla made bad decisions like this then we would still be on Joomla 1.5 series. As developers we have to drag users kicking and screaming into 2015, when so many still run 1980s websites! Anyway the robots.txt is site specific so if you dont like the defaults, change it on YOUR sites, the defaults are sane and sensible for Joomla core. Robots is, and should be, specific for each and every site after it has been set live – the defaults are correct for a stock Joomla installation after install. Dont like it, change your site, not millions.

Bulbul Hasan

Thanks a lot, i face the same problem, and i use your tips now my site is
Awesome! This page is mobile-friendly.
Thanks again for nice tricks.

Aaqib

Instead of removing components, modules etc from robot.txt just add this
#Googlebot

User-agent: Googlebot

Allow: *.css

Allow: *.js

Aaqib

Thanks Nick, this would be great help for people having similar issue. I tried that solution on my Joomla website and it worked perfect.

Michael Mason

I’ve been trying to work through this with some of my Joomla websites and still have issues with some.
I’ve tried:

• Commenting out certain directories

→ That did not work

• Adding the Allow options as described here

→ On one site in particular this does not work
Any suggestions…?

Michael Mason

Hi Nick and thanks for the response.
The site in question is:

[url=http://www.divingtenerifescuba.com]http://www.divingtenerifesc…[/url]

Michael Mason

It does. I was kind of thinking the same myself so it’s nice to have that confirmed.

Martin

Hi Nick,
Thanks for the info, I’m still really struggling to fix this issue on my site. here’s my robot.txt file…
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Allow: *.css
Allow: *.js

Martin

Thanks Nick it’s [url=http://www.cd-duplication-services.co.uk]www.cd-duplication-services…[/url]
Thanks

Martin

Done this. Still not worked 🙁
[url=https://www.google.co.uk/webmasters/tools/mobile-friendly/?url=www.cd-duplication-services.co.uk]https://www.google.co.uk/we…[/url]

Martin

Fixed!! This fixed the issue straight away. see below
To fix that problem, add this rule to the TOP of your robots.txt file, which gives Google permission to index js and css files regardless of where it finds them:
#Googlebot

User-agent: Googlebot

Allow: *.css

Allow: *.js

Martin

Hi Nick it’s www. cd-duplication-services .co.uk
Thanks

43
0
Would love your thoughts, please comment.x
()
x