Auto-Update or Die
One of the really interesting and scary things about this month’s Drupal issue is how fast the hackers moved.
Hackers moved faster than the vast majority of site-owners could respond.
It took a lot of courage for the security team to say this, but anyone whose site didn’t update between 4 pm and 11 pm UTC (London time) on October 15th should assume they were hacked. That timeframe is outside of normal working hours for everyone from Paris to Tokyo.
My guess is anyone on a big name Drupal host (Acquia, Pantheon, BlackMesh etc.) is safe because those companies rolled out fixes quickly
But if your site is on an average hosting service and you didn’t respond inside 7 hours, you may be in trouble.
Next time will be worse
So your Drupal site was updated in time?
Congratulations. You were probably using an expensive host. If not, you were probably in the Americas or Britain, weren’t being distracted by other work and didn’t have any layers of red tape to cut through before applying patches.
But what about next time? This time hackers saw the Drupal security hole and hit it very fast and hard. Next time they will likely be faster and have better tools. Next time it could be a Joomla hack inside 5 hours or a phpBB3 hack inside 3 hours.
So what can open source users and projects do in a world where attacks happen so fast? What we can do if the hackers respond faster than we can?
I believe the only solution capable of meeting this new challege are automatic updates.
2 things for open source users to do
- Choose hosts that provide auto-updates: Right now, the best thing you can do is get yourself to a host that automatically rolls out security fixes. If your host didn’t automatically update you this time, you should leave.
- Choose software that provides auto-updates: The next time you choose a new software platform, choose one that will roll out automatic security updates.
2 things for open source projects to do
- Provide auto-updates: Software projects must start to automatically push out security updates to all sites that don’t expressly disable that feature. This is absolutely the only way to guarantee that most sites are safe. WordPress has proven that this can be done. There are now no excuses. It’s not an exaggeration to say that projects now have a window of opportunity and the incentive to make this happen. Fast forward to 2016: plenty of projects will have solved this problem and it will be hard to recommend those that haven’t.
- Work with the largest number of hosts: Auto-updates are hard for projects to build and will take time to implement. In the short-term, projects could defend themselves by working with the largest possible group of trusted hosts. Perhaps they should also publicize that list of trusted hosts.
Conclusion
Auto-update or die.
That maxim now applies for both websites and projects.
On a good day, only 25% of Drupal, WordPress and Joomla sites are up-to-date. Now even the most responsible users can’t move quickly enough to keep up.
This event has changed the game for all of us. Our current solutions rely on individual site-owners and they are no longer enough. Auto-updates are the only tool we have that can keep a very large number of sites safe, in a few hours or less, across multiple hosting platforms.
Auto updating modules, and software on your server isn’t always the best idea. If a update changes some functionality how does your application continue to function with changed code or output?
We ran into a issue on one of our current server builds where auto-updating updated php, nginx and mysql. We had checked through all the server configs to see why we lost 80% performance over night. And in the end we finally went back to the software installed and started checking what updated. Turned out that each of them updated, and at this point debugging configs, which worked well already, against previous versions of php, mysql and nginx just sounded like a nightmare.
So my two cents, don’t go with auto updating. Develop a update process, keep track of security updates for software you use. When some software has a vulnerability, follow your process, TEST, and then deploy when everything works.
Thanks Chad. I would have agreed with you even a month ago, but this hack has changed my mind.
You can develop an update process and keep track of updates, but … what happens if you were in Moscow when this Drupal security issue was published? You had between 7pm and 2am to respond. In Hanoi? Between 11pm and 6am.
And next time the window will be smaller.
In short, I think this hack has proven that our current methods aren’t enough.
Steve I really agree that this is a game changer. Surely this episode could potentially do great damage to the reputation of Drupal and I have seen Drupal’s security system concerning drupalgeddon described as “shocking” and a “fiasco”. I am amazed if anyone can think that the current system of “update within 7 hours or get hacked” is adequate or fit for purpose for a professional CMS. Automatic security updates (at least for high level threats like this) with an option to opt out seems like a no-brainer. I wonder what % of Drupal sites managed to update within 7 hours and how many working hours will be lost to this? I am surprised that Dries has nothing to say about it on twitter or on his blog. Can you please come back to this subject and keep us updated about any discussion of the possibility of automatic security updates.
Hi juc56
Yes, I agree 100%.
I tried to avoid calling Drupal out in this post, because I think it’s a widespread problem. Would Joomla have dealt better with such a hack? phpBB3? vBulletin? Magento? No, they’re all vulnerable to this.
It’s a problem that’s baked in the thinking of many open source developers. We have a laissez faire attitude and prefer to put the responsibility onto our users.
If that thinking doesn’t change, most of our software installs will become botnets.
Absolutely, on continuing this discussion. I’ve already asked our developers to look at auto-updates for all our software. We hope to lead by example and keep talking about this.
Another thing is that it seems that Acquia, Pantheon etc were tipped off about this 7 days before Oct 15 and so had plenty of time to prepare [url=https://www.acquia.com/blog/shields]https://www.acquia.com/blog…[/url] But it seems that the rest of us are thrown to the wolves! Why did the Drupal Security team not forewarn us of the approaching disaster by publicising the importance of the upcoming security update for at least several days ie before revealing the details of the weakness to the hackers. I would have happily got up in the middle of the night if necessary to do the patching if I had been warned in advance about the seriousness of the situation.
Yes, that’s a valid point.
It may be that the best advice is to make your sure your Drupal site is hosted by a company with a representative on this team: [url=https://security.drupal.org/team-members]https://security.drupal.org…[/url]
> I tried to avoid calling Drupal out in this post, because I think it’s a
> widespread problem. Would Joomla have dealt better with such a
>hack? phpBB3? vBulletin? Magento?
Would WordPress???
There is a lot of discussion around Drupal adopting an auto-update strategy like WordPress. Yet AFAIK the default WordPress update check works on a 12-hour cycle. The official Drupal announcement suggested considering a site hacked if not patched within 7 hours, other advice is 3 hours. A WordPress site risks being 5-9 hours too late if auto-updating. Surely the evidence here is that flaws are being exploited faster and faster. Auto-updating may help, but it would offer no guarantees in this case. In fact, does auto-update provide false assurance?
Hey Neil
Yes, first – absolute kudos to WordPress. We wouldn’t be talking about this as the best route forward if they hadn’t proven it can be done.
Second, on the speed, that’s a great point. I guess in this case there would be several thoughts:
1) Auto-updates would still save a majority of sites (all the sites that could be fixed in 7 hours)
2) There could be a tweak to the disclosure policy. The release could be announced with the details of the vulnerability released after the auto-updates were complete.
3) In the long-run, I think the WordPress goal is to bring the benefits of SaaS to open source. That means users may never know that WordPress has updated: [url=https://www.ostraining.com/blog/wordpress-news/matt-updates/]https://www.ostraining.com/…[/url]
Hi Steve,
But that’s the point, I’m not sure the WordPress approach is the best route forward! Would auto-update in this way really save a majority of sites? Assuming attackers have already fingerprinted sites and know *which* sites to attack in advance (not hard to build a list of known sites running Drupal, WordPress, etc.), then targeting all of those in that time window seems perfectly feasible. I’ve read evidence that sites were being attacked alphabetically, suggestive of a methodical approach like this?
Likewise silent updates wouldn’t be that hard to monitor and check for differences. It’s not like the announcements in this case came with a detailed description of how to hack.
IMO polling for updates is not the fix. Some sort of subscribable push notifications to start an update handshake or alternatively put sites / servers in some form of safe mode might be, but doing that in a way that’s not also a vector for attack …
Thanks Neil
More great points again. Putting sites into safe mode is a very interesting alternative option.
I’m not welded to auto-updates as the solution, a solution of some sort is essential.
If we don’t manage this, open source projects risk becoming like Windows … popular, but also owned by hackers and the host for botnets.
In the WordPress world, I think it’s really interesting to watch Jetpack. They have monitoring already, and they’ve just purchased a security firm whose features are going to be added soon: [url=http://www.fiercecontentmanagement.com/story/automattic-acquires-bruteprotect-boosts-security-wordpress/2014-09-02]http://www.fiercecontentman…[/url] I think the SaaS-ification of WordPress is well underway with JetPack.
Thanks Nick Savov for linking me to this, and I’m glad someone brought this up. A lead developer of WordPress here, and one of the architects of our automatic updates.
I’ve sweated through, I think, 5 different auto updates, and we’ve learned a lot from each. We’ve made adjustments in both WordPress core and on [url=http://WordPress.org]WordPress.org[/url] (many at the network level) to ensure we can update as many sites as quickly and efficiently as possible.
We’ve definitely talked openly about being able to update every site on the internet within 12 hours. But it’s actually only 12 hours by default. We have the ability to instruct installs, in advance, to check for updates more frequently than usual. By doing this, we believe we can serve tens of millions of updates in under an hour.
Congrats, Andrew, on all the hard work and making it so effective.
In short, it seems fair to say that auto-updates (if done correctly) could prove to be a strong line of defense against major vulnerabilities.
I still don’t agree with a general statement that every project should auto update all the things. Unless there is going to be an entity within the project that assumes responsibility for an update they push taking your site down or causing data or revenue loss, the project should not have auto updates. Which is why IMO it would be a very bad move for Joomla to even try it, the resources just aren’t present to deal with that scenario. WordPress can do it and I have it on good info that failed updates are followed up on by a team (wasn’t made clear whether that’s volunteers or an Automattic team, but that’s irrelevant to the fact the followup is there).
Even just auto updates for security issues is a risk; the security issue patched with Joomla 3.3.5 broke the update component and though we were able to limit the damage based on the time we had that update posted on the update system, if we had been auto-pushing that update during that same period the numbers would have been much larger.
Hey Michael,
Yes, it’s absolutely a challenge and one that’s difficult for everyone to meet right now.
Short terms? I guess the best solution is to work with the hosting companies, firewall providers and so on to make sure they’re ready and able to apply patches.
Long term? It may be OSM needs to make money available in this area. It may be that better tools will emerge in the months to come to handle auto-updates.
But long term, I have hard a time believing it isn’t an essential problem.
It doesn’t have to be either or… auto-update should be available but something that’s configurable so that it also fits into process. One site owner may want to enable it to protect against threats automatically and then go back and test post updates. Yet another site owner may opt to disable it and test first. Auto-update should exist, but it should be a configuration choice as to how to use it.
Yes, agree 100%, Heath.
Auto-updates are essentially a fallback option for users without professional grade resources.
In this article Acquia, Pantheon and BlackMesh are mentioned as high end Drupal hosting that updates, secures and supports down to the Drupal layer. Can someone point me to a similar service for Joomla? IOW: optimized joomla support and hosting.
Hi William
The best is probably SiteGround: [url=https://www.ostraining.com/weblinks/siteground/blog/]https://www.ostraining.com/…[/url] They’ve got a big set of Joomla-specific tools. They’re also a lot cheaper than all those Drupal hosts 🙂
I like Siteground but I’m talking about even more full service. My company deploys Joomla websites for our clients. We do everything. We develop the site, write the content. We host the sites, we secure the sites, we track extensions versions and update as needed. We run CloudFlare etc. I am trying to screw down our hosting and support services and costs. I’m looking for examples of other companies doing similar things.
There’s nothing quite as enterprise ready as Acquia and Pantheon, I’m afraid.
Perhaps tools like Watchfulli and MyJoomla might help?
To solve exactly the problems you mentioned in this blog post as well as in the related comments, there is a system that automates updates for Drupal and does it for other CMS soon. The service is available at [url=http://drop-guard.net]http://drop-guard.net[/url]
Drop Guard integrates with individual development and deployment platforms to ensure quality (depending on the criticality of the update).
I give personal demos and free access to test it yourself before the official release in september. I am very excited about critical feedback.
[quote=Neil C Smith]Hi Steve,
But that’s the point, I’m not sure the WordPress approach is the best route forward! Would auto-update in this way really save a majority of sites?[/quote]
You’re missing an important point. Drupal is a nightmare to update compared to WordPress, meaning far less people will update Drupal sites. So never mind, the ‘window’. I’ve just seen how difficult the Drupal update process is. For this reason, I’m going to spend the day persuading a client to move their website to WordPress (their current website is in D6, and I was asked to take over). The previous developer didn’t do ANY updates… maybe because they are so damn complicated and annoying.