Security Patch Overload

The Endless, Thankless Cycle of Patching

How many of us responsible for managing security patches feel a great sense of accomplishment and gratification when we install a bundle of updates, knowing that on the horizon, a new version is about to be released? When was the last time management gave out praise on a job well done for applying patches?

This endless cycle may leave many asking themselves:

• Why?
• Is there a better way?
• How can we improve this process?

Individuals tackling this patching feat should potentially have a more dramatic title such as "Risk Mitigation Heroes" or "Cyber Crime Stoppers." The importance of patching has never been more critical to the security of any organization, especially in today's environment, where the number of security-related patches continues to increase daily.

Digging into the "Why?"

Most of us try to keep our bodies healthy by exercising, eating right, washing our hands, getting enough sleep, etc. We do this to limit the ability of viruses and illnesses to compromise our immune system. A well-structured patch management program could be considered in a similar way. We install patches to reduce the number of attack vectors that could lead to a compromise of our infrastructure and data. In reality, we're keeping our infrastructure healthy.

We hear of organizations getting compromised on nearly a daily basis. Many of these breaches - approximately 60% are caused by cyber thieves taking advantage of missing security updates, oftentimes in conjunction with social engineering to exploit known vulnerabilities. Up to 60% of hacked small and medium-sized businesses go out of business after six months.

The challenge of patch management isn't getting easier:

• Patch volume is increasing. In 2020, there were 18,358 CVEs (common vulnerability and exposure) introduced; that number increased to 20,194 in 2021.
• Identifying when patches are available. Most organizations have many different types of IP-connected devices of which any could have vulnerabilities.
• Testing can be difficult. Especially with smaller organizations, it’s very difficult to have a test environment with the applications and interfaces needed to properly test patches before deploying.
• It's time-consuming with limited staffing. This is a common theme in most organizations.

Building Your Patch Management Program

There are many factors to consider when developing your patch management program. One of the top priorities is getting management of the organization involved with the development and documentation of the program. Getting their full support and commitment to the program is vital to the success of the overall patch implementation.

Depending on the organization's size, the patch management role should clearly be defined and assigned. Larger organizations with available resources should have a dedicated team assigned this responsibility. Many smaller organizations may use a managed service provider (MSP) to manage updates.

A modern patch management program should address the following topics:

• Inventory Your Devices:
  All IP devices should be included in your patch management program. Many organizations think patching only entails PCs, laptops, and servers. In reality, any device connected to your network could introduce vulnerabilities. Consider the IoT (Internet of Things) and the vast number of devices that covers, including smart TVs, IP surveillance cameras and DVRs, smart speakers, appliances, thermostats, and more. Each of these devices introduces the potential for additional risk.
   
• Scope Your Patches
  All applications should be included, not just Microsoft products. Nearly every organization uses a wide range of applications, which may include Adobe, Java, Oracle DB, and more. Don't forget about your network devices that likely require firmware updates a few times a year, including firewalls, printers, routers, and switches.
   
• Patch Frequency
  Timing, prioritization, and Service Level Agreements (SLAs) should also be addressed in your patch management program. The average time it takes between identifying a vulnerability and the appearance of an exploit in the wild has dropped by 66 percent, from 45 days to 15 days over the last decade (TrendMicro). It's important to have guidelines defined that state how quickly patches will be deployed once released. The timing should also take into consideration the criticality of the vulnerability to determine the deployment priority. Many organizations use the Common Vulnerability Scoring System (CVSS) to determine patch criticality, but every organization needs to evaluate how that may apply to their infrastructure.
  
  Additionally, you don't need to apply patches to everything within the same timeframe. Use your IT risk assessment to determine the importance of a device (or group of devices, like workstations) or application, and use those ratings to create categories (or tiers) of IT assets with different patch frequencies. For example, your Tier I (most important) assets may have a time-to-patch frequency of 15 days, while Tier III (less important) may have a time-to-patch frequency of 90 days.
   
• Testing Patches Prior to Deployment
  Testing your patches is perhaps one of the top challenges of many organizations. With limited staff and resources, how can proper testing be performed and yet still meet the deployment SLAs defined? Every organization should have dedicated testing resources to address the potential negative impact patches may introduce. No one wants to have to take production devices offline because of a failed patch, so plan ahead and test before deploying patches and updates.
  
  Depending on the size of the organization, testing resources might include a few non-critical production devices or an entire lab with all the different devices and applications used throughout the enterprise.
   
• Patch Roll-Back
  Anyone that has performed patch management has experienced a patch gone wrong. In today's complex networking environment, the chances that a security patch negatively impacts your users' ability is substantial. Make sure you have a plan in place to undo certain patches that cause either usability or connectivity issues, allowing users to get back to their daily job duties quickly.
   
• Cloud/Web Applications 
  Devices or applications that are hosted somewhere other than your physical location pose another challenge to the patching process, particularly as more applications migrate to a software-as-a-service (SaaS) platform. How do we ensure proper patching? Typically, when a third party is hosting the application, it's important to include patching responsibilities in the contract and define the responsibilities, including SLAs.
   
• Patch Automation
  The process of applying patches needs to be done in an automated fashion with detailed reporting. Patching automation requires the use of a patch management solution, for which there are many well-respected and proven vendors offering robust systems (some of which are even cloud-based). Keep in mind that you need a solution that can address all applications, not just Microsoft products. A monthly patching report should be provided to management, communicating the status of the overall process and illustrating the potential risk exposure.
   
• Patch Validation (Testing):
  Here's where vulnerability management comes into play. It's important to validate that known vulnerabilities have been remediated by your patching process by performing a vulnerability assessment (VA). There are many vulnerability scanning tools available that can automate this process and scan all devices on your network, potentially even your cloud applications.
  
  When performing a VA on your internal devices, it's important to perform a credentialed VA that fully evaluates your PCs, laptops, and servers. While an uncredentialed VA can still provide some value, completing a credentialed VA, which utilizes domain administrator credentials to fully scan the hard drive and registry of devices, performs the true job of a vulnerability assessment, which is to find ALL the vulnerabilities on your network, not just some of the vulnerabilities.
  
  Another patching challenge, as seen with several Microsoft KBs (ex. KB 4022715), includes updating the Windows Registry, along with deploying the patch, before the vulnerability can be fully remediated.
  
  A monthly or continuous VA is recommended to assist your organization in knowing where you're at regarding today's known vulnerabilities (there will be more tomorrow). Results from your patch testing process should be reviewed regularly, and a report should be provided to management that quantifies how effectively your organization is performing patch management.

Take-Aways

Security patching is all about managing risk by reducing the attack vector to which the organization is susceptible. Patch management is critical to the health and survivability of any organization. Performing regular patch management and validation of patches moves you into the "we're proactively managing our risk” category from the “we don’t know what our risk looks like” mindset. 

All organizations should periodically assess the effectiveness of their vulnerability identification, remediation, and response process, making sure it includes the items mentioned above. 

We have the tools and the capability to drastically reduce the risk of a data breach, network compromise, or ransomware attack. While it costs us some time and a fair number of resources, it sure beats the alternative.