Patch Management – Essential Infrastructure Maintenance

By Aaron Arlotti

Your vehicle needs proactive maintenance to keep it running efficiently as well as to extend the overall life of the car. You do this by regularly changing the oil, rotating the tires, and replacing the brakes. Ignoring this required maintenance leads to costly repair bills and could possibly lead to an accident-causing injury.

The same type of maintenance is required for the computers and your network infrastructure on your companies’ network. Patching and proactive maintenance plays an essential role in ensuring the security and stability of technology systems.

In this article, we will provide a brief overview of what patching entails. We will discuss some of the risks associated with patching and explain why the routine patching of systems is a critical practice.

The Process

To keep things simple, we will generalize a bit and forgo discussing the distinction between the various patch types you may have heard of.

Although patching procedures vary slightly from one device or system to the next, the process is generally the same: Inspect, Download, Install, Re-boot, Repeat. Some circumstances require the added step of making a backup before starting. Others require manually re-starting applications and services when finished. At times the computer, server, or network device will also need to be restarted before the changes can be applied. While it may sound complicated, this is a straightforward process for highly trained IT professionals who understand the technology.

That said, there are always those rare, dreaded occasions when patching causes unintended problems that require additional time and effort to remediate.

The frequency that the patches need to be applied can vary depending on the release cycle of the vendor. Except for critical vulnerabilities, patches are routinely released weekly, monthly, or quarterly. Critical vulnerabilities need to be patched immediately and are released on an as-needed basis. These critical patches could be likened to a safety recall you might receive on your vehicle. With safety compromised, it is critical that the repair be installed as quickly as possible.

The National Institute for Standards & Technology (NIST) suggests that “ideally, an organization would deploy every new patch immediately to minimize the time that systems are vulnerable to the associated software flaws.” At the same time, it acknowledges the realities of timing, prioritization, and testing as intertwined issues.

Example Scenario

Since it would require a lengthy treatise to cover the multitude of patching scenarios, we will use the application of operating system patches for a basic Windows server as our example. During patching, the server’s operating system is analyzed to identify and download new patches released by the software manufacturer that have not yet been installed.

In this example, Microsoft may release new patches as often as several times in each week. Because some patches must be installed as pre-requisites to subsequent patches, they are often broken into groups to ensure they are installed in the proper order.

The act of downloading and installing updates is rarely disruptive, with the exceptions that downloading anything from the Internet uses bandwidth, which can diminish speed, and servers that are either old or already burdened by a heavy workload can occasionally be noticeably slower during the process of downloading updates.

After each group of patches has been installed, it is almost always necessary to perform a re-boot of the server. Re-booting is a technical term that boils down to basically allowing the operating system to shut down properly and then re-starting the system. When a re-boot is initiated, any devices electronically connected to the server will be disconnected at that time. For this reason, it is essential that users are informed in advance of maintenance windows. This allows them an opportunity to close client-server applications and mitigate the likelihood of losing work in progress.

Whenever re-boots occur following the installation of patches, it should be expected that the process of re-starting may take longer than usual. This is because updates often require new system files from the patches to be written to the proper places before the re-boot completes.

When finished, service is restored. Users should be cautioned however, that the fact a server has re-appeared online does not necessarily mean the maintenance cycle has been completed. If the server needed to install prerequisites, the entire process must be completed again, sometimes multiple times, to ensure all available patches have been properly installed.

When Things Go Wrong

We have all had the experience of that mysterious and odd noise coming from our vehicles after having a repair completed. While we are confident in the skills of the service professionals that handle our cars, there are times when parts may be faulty, even when sourced from a reputable vendor. After-market additions to the vehicle may not be compatible with the factory parts that have been installed.

Similarly, whenever patches of any type are installed, there is always a degree of risk that must be understood and agreed upon by the client and the installer. Despite the best efforts of third-party software manufacturers to test updates in different environments, it is not always possible for them to avoid mistakes. This reality presents a difficult challenge for IT Professionals.

As implementers, it is the IT Professional’s duty to deploy updates rapidly to promote security and stability. Should he or she elect to forego installing updates for any significant period, there is a risk of being held responsible for a resulting data breach. At the same time, IT Professionals understand that whenever updates are installed, there exists a small, yet very real possibility that doing so could create problems. While it has been rare, we have experienced a wide range of unforeseen and unwanted instances that vary from minor software conflicts and sluggishness, to network-wide outages. Further exacerbating such occurrences is the fact that while some side-effects can be identified quickly by the IT Professional, others may be intermittent or remain hidden until discovered later by users.

To put it simply, IT providers are at the mercy of the software developers that deploy the patches. We do not have direct access to the code of the updates they are deploying, nor the context in which any changes it makes will adversely impact the client. Patches and updates are installed as an industry practice and rely on trusting the good-faith representations of software providers that their updates will be beneficial and (to the extent possible) not unintentionally harmful.

When update-related problems occur, those affected who are unfamiliar with the ever-present tightrope the IT Professional walks may (understandably) wonder why he or she installed updates and “caused” the problem to begin with. After all, everything seemed to be working fine before. Conversely, the opposite question would be asked if network security was compromised and a computer or the network were breached.

Tech-savvy clients often rightly ask a more targeted question: Why didn’t the IT Professional test the updates within a subset of the company’s environment first before deploying them company-wide? This question is reasonable, and the answer is most often because of cost. The frequency at which updates are released can be quite considerable, in some cases as often as several times a week.

While it’s not uncommon for very large corporations to employ some manner of internal testing department and pre-approval process for patches before they are deployed, the cost of time and equipment for most businesses to do so far outweighs the added expense associated with remediating the occasional patch-related problem.

Cost notwithstanding, as a provider of Concierge Level® IT support, Sandbox Technologies does offer custom-tailored testing and deployment services for those clients that wish it. While we would caution that it is most often determined to be cost-prohibitive, interested parties are urged to contact their Sandbox Technologies Account Manager for a professional assessment as estimate of costs, and options available to briefly delay patch deployment providing an opportunity to see if users of specific software or solutions report problems in various online forums.

The Conclusion 

Despite the risks, patching is an essential maintenance task that must be undertaken regularly. If you would like more information about software updates, security patches, or Patch Management, please consult your Sandbox Technologies Engineer, Consulting CIO or Account Manager.

Aaron Arlotti is Manager of Remote Support operations for Sandbox Technologies.