Can You Trust BuiltWith, W3Techs and Software Usage Statistics?
The Drupal Association put out a press release today, quoting some statistics about Drupal.
The press release said:
- According to Drupal.org stats, Drupal now powers more than 1 million websites.
- According to Builtwith Research, Drupal powers more than 12% of the world’s top 100,000 websites.
I regularly see stats like this fly around the web, particularly on Twitter.
So, I decided to use these Drupal stats as an example and answer the question: “Can you trust software usage statistics?”
Sources of usage statistics
There are three websites that collect very large amounts of information available about Drupal:
- Drupal.org
- W3Techs.com
- BuiltWith.com
There are some smaller sites such as Wappalyzer and TopDrops, but I excluded those because either have much smaller data samples than the big three or didn’t present their data in ways that could be easily compared.
Out of these three sites, not all the information is equally valuable. It’s worth giving priority to sources who are closest to the actual software installs. Because Drupal.org connects directly to actual Drupal sites, its data is probably the most reliable.
I would give the least credence to BuiltWith. Both Drupal.org and W3Techs publish their methodology, but BuiltWith has only the following, empty statement:
“Every website gives off “signals” that they are using a particular technology. We can track these signals and determine if a site is using a particular web technology from it.”
- Drupal is used by 796,045 websites according to BuiltWith.
Answer: These two numbers agree fairly closely, so we can estimate that Drupal is used by close to a million websites.
Is Drupal usage growing or shrinking?
- Drupal use is trending up slightly (from 800,000 to 934,000 sites during 2013) according to Drupal.org.
- Drupal use is trending up very slightly according to BuiltWith.
- Drupal use is trending down very slightly according to W3Techs (Update, that may be to a change in the W3Techs methodology).
Answer: Taking an average of the three sites, Drupal usage is trending slightly up.
What percentage of the web is using Drupal?
- Drupal is used by 0.3% of all websites, rising to 3.9% of the largest 10,000 websites according to BuiltWith.
- “Drupal is used by 5.4% of all the websites whose content management system we know. This is 1.9% of all websites.” W3Techs
Answer: This is all over the place, but the midpoint of these averages is just over 2% with a noticeable lean towards large sites.
Conclusions on the Drupal press release statistics
So what’s the verdict on the Drupal press release?
- Yes, Drupal probably does power around a million websites.
- The 12% figure is odd. I’m not sure where it came from (BuiltWith says 3.9%). Even if accurate, it doesn’t reflect the other statistics on the percentage of the web using Drupal.
Conclusions on usage statistics in general
I started this exercise as a sceptic but was pleasantly surprised by the fact that could extract some general agreement from these different sources.
There were at least three points where we could get some consensus:
- The total number of sites
- The general trend in usage
- The general market position
The one point where the results were radically different was the % of the web using Drupal. I wouldn’t feel comfortable quoting any of that data. I suspect that BuiltWith and W3Techs have radically different ideas about the size of the internet.
However, it’s worth remembering that even the closest answers differed by at least 25%. The data gets even more wild when we start to include other sites such as Wappalyzer.
If you’re doing this research, always get at least two sources of data, and if possible three or more.
Hey Steve, if you notice the dip on W3Tech’s usage between May and June of 2013, there is a press release on their blog that states they started using a larger sample for their numbers.
Basically, W3Tech started out sampling 1 Million sites in the beginning, to gather their stats. Then, mid 2013, they moved to a 10 Million site sample for their data. Both WordPress and Joomla received a boost to their overall usage numbers when this occurred, however, Drupal’s usage number dropped a bit. Further enforcing the premise that Drupal tends to power the upper end of the market share.
I have 16 Month’s worth of monthly stats breakdown on Joomla using W3Tech and Builtwith as my basis. Granted, the numbers may not be exact, but used as the basis for statistical data over a time period, show growth or decline over that same period.
Thanks for the write up on this!
Thanks Robert. That is useful to know.
So w3techs changed their sample size 10-fold and caused a noticeable change in all their results … but don’t note that on any of the graphs.
That kind of fact should increase our skepticism further.
Very interesting post, Steve. Datanyze shows Drupal with 2,868 websites in the top 100k websites based on Alexa rank: [url=https://www.datanyze.com/market-share/CMS/?selection=1]https://www.datanyze.com/ma…[/url]
Thanks Jon. Your site proves the articles’s point to some extent.
Your own site is showing 2.8% for the top 100k sites. Compare that to 3.5% for BuiltWith.
Those numbers agree about as much as we could hope for.