Blog
Patching stupidity: Why being in a hurry can break your system!
And how you can easily prevent that
Last week we all witnessed how much damage can be caused by rolling out a patch too quickly. I hope you've only seen it from a distance, but a lot of you must have been victims of the calamity. CrowdStrike rolled out a buggy patch and many of CrowdStrike's customers must have installed it without noticing. At least I can't imagine that patch would have passed a rigorous testing program.
The short term consequences
An almost complete disruption of air traffic, hospitals unable to work, banks that can't execute payments for their customers, retail chains that can't plan their employees, big and small companies that need to close their doors for hours or even days. For many victims a blessing in disguise that it happened just before a weekend, that gives them the weekend to clear the rubble. For airports, hospitals and banks the misery continues for days. even though the problem is less severe than on the the day the world stopped turning for them. Some passengers are still stranded ...
The long term consequences
I don't have a crystal ball, but I don't need one to know that a long legal battle is coming. CrowdStrike will change it's name once they're out of the spotlight. This is also something to think about if you'r on the brink of buying a Microsoft vendor lock-in by moving all your digital belongings to Azure on a contract for 3 years or more.
There's a lot of bad patches
Just looking at Microsoft's “Patch Tuesday” it happens quite often that a patch must be pulkled back because it's faulty. Not every month, but way too often in my opinion. And it's not just my opinion that counts ... Sometimes it hits a certain group of users, sometimes all Windows customers are duped. Quite often it's a 'blue screen' problem, in other words, a PC rendered completely unusable. And when your PC is inoperative you first need to know which patch is the bad apple and then follow a long process to remove it.
In that light it's been a really bad move by Microsoft to force most Windows users to install patches On Windows 10 and Windows 11 patches are installed without consent from the user or the owner of the computer. Bad patches are installed on your Windows machine quite often, with no possibility of stopping the installation, even if you know the patch is not OK and will cause problems on your PC.
And that's just Microsoft. A lot opf other vendors install updates without your consent. Google does that with Chrome for example, and who doesn't run Chrome, if only as a second browser. Global Stats StatCounter attributes a market share of almost 66% to Chrome ...
If it ain't broken, don't fix it
I have a Logitech mouse and keyboard. Regularly Logitech sends me a notification of a 'critical' update to my drivers. On closer examination the patch almost always just supports new hardware. Hardware I don't have. As the existing driver is working flawlessly on the hardware i do have the update is unnecessary for me. It's even a significant risk. The existing driver works without problems and the new one can contain a bug. As it's a driver, potentially a bug that makes the machine unusable.
You may consider that 'critical thinking'. And it is. In my opinion, more people should think critically. With other software you should also ask yourself if really need the latest update. What did the really change in the new release? And is a nicer button worth the risk that your PDF reader stops working? Or your email client?
A lot of suppliers these days tell you you must install all patches as soon as possible. And i vehemently disagree with that. For critical security patches it's necessary to install them with controlled haste of course, assuming they've been tested if you have the means for that. And other security patches should not be passed over either. But all other patches should be critically examined to determine if you need them. And if they fix a problem you don't have anyway or add functionality you're not going to use, I'd be very hesitant with their installation.
Test your patches
Business users usually have more possibilities to prevent patches being forced on them. With certain configurations they can enforce that patches are not installed without prior consent. And they can use software that install those patches automatically after they've been tested and approved by the IT department. Because software often contains - an more often than we know - bugs en with proper testing you'll find them of course.
That you shouldn't rely on supplier testing for the full 100% was demonstrated by CrowdStrike. An Microsoft also confirms that several times a year. You can solve that only by doing your own testing.
"But Hugo, what if there's a big security leak, that hackers are already abusing?" I hear you ask. For those situations you need accelerated test procedures that only examine that patch for that specific leak closely. Such patches are usually made by suppliers in great haste (of course, the leak must be fixed NOW), with a shorter test procedure, and that gives a higher probability of errors than a regular patch does. If such a patch brings down your systems you're further up the creek with no paddle than if a vulnerability remains open a few hours longer. By the way, you can often avoid such a vulnerability temporarily by adjusting your firewall or by turning off certain software components. There are so many ways to avoid risks.
For small companies and consumers this is not an option unfortunately. Where possible it's wise to delay the installation of patches somewhat. That will give you a bigger chance that someone else trips over the bug and the patch gets pulled before it's installed on your PC. Automatic updates appear convenient, but they can be real trouble makers..
The problem is not always in the system you're patching
Different systems tal to each other. If one of them gets patched and the other isn't, you could lose connectivity for example. Newer SSL libraries no longer support older protocols, for security reasons. And if you're still running some legacy machines or legacy software (and who doesn't have legacy) your patched machine will no longer communicate with the legacy. Very safe. Absolutely undesirable.
I have a lot of similar examples. The moral of the story is that you have dependencies in your application landscape and you should include them in patch testing. Especially if you're patching central components like your active directory server.
The real problem often remains invisible
Not all bugs bring down your system immediately. A less apparent problem is that a patch slows down your system significantly. We see that at Sciante customers regularly. We've seen that with the upgrade from SQl Server 2012 to 2014. And with the BIOS patches for Spectre en Meltdown. It may not look like a big issue, but how much work time is it costing you? Sometimes systems are slowed down by tens of percents. You at least want to know that it's happening and how big the impact is.
Consequences are employees slowed down in their work, but don't forget about rising cloud cost if a system is less efficient. Exactly what you don't want and an increasing number of companies and organizations is finding their way to our door to deal with these problems once and for all, because, fortunately, all this is avoidable.
How do I stay in control?
An important step is proper monitoring. Including in your test or acceptance environment where you test your patches. And keeping tabs on the monitoring. If you check your speedometer only occasionally, a speeding ticket is a likely reward. With proper monitoring you know the situation before and after the patch. And you can compare them to see the influence of the patch on performance.
Need help? Book a 15 minute zoom call (or a bit longer) with me and you'll know how to stay ahead of problems. Don't let disruptions surprise you and book your call today.