The Critical Elements of the Patch Management Process

“After the flames from the slammer’s attack were doused and the technology industry caught up on its lost sleep, we started asking questions. Why did this happen? Could we have prevented it? What can we do to keep such a thing from happening again?”⁶

These are some of the questions that the security industry asks after every major security incident. Today most security incidents are caused by flaws in software, called vulnerabilities. It is estimated that there are as many as 20 flaws per thousand lines of software code.¹ Computer Emergency Response Team/Coordination Center (CERT/CC) statistics reveal that the number of vulnerabilities reported has increased dramatically over the years, from only 171 in 1995 to 8064 in 2006. Along with vulnerabilities, the sophistication of attack tools has also advanced over time. Using the interconnected nature of the Internet and automated attack tools, attackers exploit software vulnerabilities at an alarming rate to cause serious damage to organizations.

Although the ultimate solution to fix software vulnerabilities is application of patches, until a few years ago the term “patch management” was not in the general vocabulary of even the most advanced information technology staff. Today, “patch management” is not only in the common vernacular of most IT staff, but it is also one of the most essential responsibilities of IT departments. Security threats stemming from the exploitation of vulnerabilities pose serious risks to corporations, including unauthorized access to systems, corruption or modification of data, and unavailability of system resources to authorized users. Systematically applying patches to vulnerable systems through effective patch management can effectively reduce the number of security lapses. It is estimated that 95% of security breaches could be prevented by keeping systems up-to-date with necessary patches.⁷ Though recognized as important for security, many organizations do not have a clear understanding of the elements of patch management and how these elements impact the success of the patch management process.

Back to Top

Patch Management Process

Patch management cannot be viewed as simply a necessary activity or a product to apply updates. Patch management is multifaceted. It is a process that consists of several critical elements that together contribute to the success of the process itself. Effective patch management practices have been identified in literature across the technology and security sectors, including the federal government, patch management software vendors, and other computer security experts,¹ and it is interesting to note the recurrence of a common set of elements. Critical elements of the patch management process include the following:

• Senior executive support. Senior executive support is management’s recognition of information security risk in the organization. But beyond recognition of the problem is management’s support of the patch management process, including ensuring that appropriate resources are directed toward the effort across the organization.
• Dedicated resources and clearly defined responsibilities. Dedicated resources and clearly defined responsibilities are important to the success of the overall process. Staff must be tasked with the responsibilities of defining, implementing, and managing the process.
• Creating and maintaining a current technology inventory. A current technology inventory is essential to any patch management process. A current inventory of hardware and software helps the group responsible for patch management determine the number of systems that are vulnerable and the patches required. An inventory also helps the staff locate computers and their owners across the organization.
• Identification of vulnerabilities and patches. Identification of vulnerabilities and relevant patches is important and unique to each organization’s patch management process. With the current technology inventory in hand, the responsible group can monitor for vulnerabilities and patches for software used throughout the organization.
• Scanning and monitoring the network. Pre-deployment scanning and monitoring of the organization’s network can help assess risk levels. Software tools can help identify the patch level of software on workstations so that effective remediation steps can be taken.
• Pre-deployment testing of patches. Testing patches in a controlled environment prior to deployment is a proactive step often overlooked by many organizations. Testing is important to ensure that patches function as intended and to see any potential adverse affects on an organization’s systems.
• Post-deployment scanning and monitoring. Scanning and monitoring the network specifically after deployment of patches is a significant step to ensure that patches have been effectively applied. For some sectors, such as government and health care, post-deployment network scanning can be used as an audit tool to help ensure compliance with defined standards.

Even the federal government has taken steps to address security vulnerabilities that affect systems in its agencies. IT has formalized the patch management process through the Federal Information Security Management Act of 2002 and the National Institute of Standards and Technology (NIST) has published a handbook entitled “Procedures for Handling Security Patches.” However, attitudes about the patch management process vary across sectors.

Back to Top

Survey

We conducted a survey of IT professionals in the public sector, higher education, governmental, healthcare, and other areas to determine the importance of these critical elements in the patch management process. Respondents were asked about patch management practices in their organization, as well as to provide an estimate of success rate of the process. The results of the survey provide insights into how organizations view critical elements in the patch management process and whether the type of process affects their approach towards patch management.

Respondents were instructed to consider only the workstations in their organizations for which they had responsibility. Servers and specialized machines are considered to be special cases and were not considered in this survey. Respondents were also instructed that, for the purposes of this survey, “enterprise operating system patch management” refers to “the process of applying operating system patches and updates to the computers in an organization.” Respondents were asked about the type of patch management process used in their organization using the following definitions:

• Manual: Patches and updates are applied manually at each workstation.
• Windows Automatic Update: Patches and updates are applied using Windows Automatic Update in a completely automatic mode, with no user intervention required.
• Automated: An automated patch management software product (such as SUS, HFNetChk, BigFix Enterprise Suite, and PatchLink Update) is used for patch management.

Back to Top

Survey Results

Of the 114 respondents to the survey, 42.9% were from the corporate sector, 38.6% from education, 9.6% governmental, 1.75% healthcare, and 7% classified themselves as “other.” Respondents were predominantly IT staff (47.8%) and IT management (37.9%), with the rest identifying themselves as corporate management or “other.”

As highlighted in Figure 1, the use of an automated patch management software product is most prominent among organizations, with 64.4% using an automated patch management software product, 18.2% using Windows Automatic Update, and 16.5% applying patches manually.

The corporate sector uses an automated patch management process most widely, followed by governmental, healthcare, and academic sectors. It is interesting to note that no governmental agency or healthcare institution responding to this survey use Windows Automatic Update, but rather use automated processes exclusively. We should also take note that among academic institutions, the use of an automated process and Windows Automatic Update is nearly the same, at 45.5% and 43.2% respectively. We are not surprised at the exclusive use of automated patch processes in the governmental and healthcare sectors given the more stringent regulations placed on institutions in these sectors. These sectors likely have more resources available to implement and maintain those processes with the appropriate staffing. At the same time, many academic institutions likely have fewer resources available to implement and maintain automated processes, causing them to fall back on the operating system’s built-in automatic update.

Based on respondents’ ratings about the importance of each element on a 7-point scale ranging from “most important” (7) to “least important” (1), we calculated the average importance score for each success factor to further compare the critical elements. Some interesting results emerged from this analysis. Table 1 shows the average scores of importance and standard deviation of scores (in parentheses) for each of the critical factors for all respondents as well as respondents grouped by patch management process. The minimum score was 1 while the maximum score was 7 for each factor, except identification of vulnerabilities for which the minimum score was 2.

It is interesting to see that the average scores of importance for the seven factors are clustered into two groups when compared by patch process and across all respondents. The difference between the average scores of factors in these two clusters is statistically significant across all respondents, as well as for each process type (p-value<0.0001). Identification of vulnerabilities, network scan pre-deployment, and dedicated resources rank highest in importance, while the remaining 4 factors consistently rank below the first three. The remaining four factors also vary widely in their average scores of importance across patch management processes. This grouping is likely a result of the resource-intensive nature of several of the four lower-ranked factors. Maintaining a technology inventory and pre-deployment testing of patches are activities that require significant time and effort to perform, often by dedicated and highly technical personnel. Many organizations may not have the resources to dedicate to these activities. As a result, these organizations may place lower importance on these activities simply in spite of their actual overall importance.

It is at once surprising and disappointing to note the lack of importance that respondents placed on senior executive support. Management approval and involvement is important to the success of any security activity because management dictates an organization’s security posture.² To see such low importance placed on senior executive support likely indicates that IT staff are implementing patch management processes independent of management input rather than as the result of a management mandate. However, we are likely to see an increase in management’s interest and involvement in security as a result of an increased federal regulatory environment. FTC regulations originating from Title V of the Gramm-Leach-Bliley Act make corporations and corporate officers responsible for the protection and privacy of personal information. Similarly, the Sarbanes-Oxley Act, which improves the accuracy and reliability of corporate disclosures, makes CEOs and CFOs personally accountable for violations. An organized patch management process can help corporations demonstrate due diligence in the areas of data security and privacy.³

Organizations that use an automated patch management tool perform pre-deployment scanning and post-deployment scanning more than organizations that use other processes. This is reasonable given that automated patch management tools generally provide scanning and reporting capabilities, which could also be a testimony to the importance of using an automated tool. But all organizations, regardless of the patch management process used, place a relatively high importance on pre-deployment and post-deployment scanning. These results indicate that, even among organizations using Windows Automatic Update, the desire to know the current state of security and the identification of vulnerabilities is important. These results also indicate that organizations can benefit from enforcement tools that are used to detect violations and analytical tools that are used to monitor the working environment, generate reports, and possibly predict future trends.⁴

Our results also show that most organizations test patches before deployment in their production environments, however pre-deployment testing is least common among educational institutions. More than half of the educational institutions do not perform pre-deployment testing. This is likely due to the resource-intensive nature of testing, and many of these organizations may not have the staff resources required for these activities. In addition, educational institutions that use Windows Automatic Update typically do not perform pre-deployment testing. This can be explained by the “hands-off” nature of the Windows Automatic Update process which promotes a “set it and forget it” approach to patch management. Pre-deployment testing could be performed in an environment using a fully-automated Windows Automatic Update process through the use of group policy in a network environment to control the timing of the updates.

All of the corporate respondents that reported that they do not test before deployment use an automated patch management tool as their pre-dominant patch management process. This is a typical mistake in process implementation as the automated tools do not substitute for testing. In fact, automated tools can help organizations add pre-deployment testing into the process because patch deployment is controlled throughout the process. Mistakes such as these can be mitigated through a systematic understanding of the various processes and how they can be used within an organization’s environment.

Back to Top

Effectiveness of the Patch Management Process

For a whole host of reasons, it may never be possible to attain 100% effectiveness in any enterprise patch management process. However, one determination of the effectiveness of a patch management process is the percentage of patched machines in an organization’s environment. In order to assess the effectiveness of respondents’ patch management processes, the survey asked respondents to identify the percentage of patched machines at any given time in their environment over the 30-day period prior to answering the survey.

Not surprisingly, the highest effectiveness is achieved among organizations that use an automated patch management process. Of the organizations using an automated process, 31.1% report having 95–100% effectiveness, 35.2% report having 85–95% effectiveness, and 17.6% report 75–85% effectiveness. Likewise, the least effective organizations are those that use a manual process. Among organizations using a manual patch management process, 38.9% report having fewer than 50% patched machines at any given time. The effectiveness of Windows Automatic Update, while better than the manual process, falls short of the effectiveness of an automated patch management process.

Although automated patch management tools can improve the efficiency of the update process and therefore reduce operational costs, there are also challenges associated with implementing automated tools. First, most of these utilities are very complex and partially effective.⁶ An administrator must still visit each machine if the automated installation fails. Second, these tools cannot make up for testing.⁸ Security administrators still need to test each patch internally before deploying to the enterprise automatically using these tools. Third, while some automated tools (such as PatchLink Update) provide support for multiple platforms such as Windows, Linux, and Solaris, others (such as HFNetChkPro) focus only on a specific platform such as Windows.⁹

Organizations may find that they need to include users in the patch management process. Even in organizations that use automated patch management tools, users with notebook computers or home telecommuters will not be a part of the automated process. Addressing these systems is especially important for organizations that allow telecommuting because a home system can threaten the security of the organization’s network.⁵ These organizations should train users how to use the operating system’s built-in update processes, such as Windows Automatic Update, and publish guidelines outlining the proper use of these tools.

Back to Top

Conclusion

From the results of the survey, we can see that several critical elements of the patch management process are important to all organizations. While all of the critical elements are important, we can see that several consistently rate high among organizations regardless of the type of patch management process in use. However, we find that some other elements such as senior executive support and testing prior to deployment vary widely across organizations.

The challenges will be ensuring that an organization’s patch management process is covered by appropriate policies and procedures, has adequate resources dedicated to its execution, and has the proper tools to effectively monitor for vulnerabilities and provide reporting for remediation.¹ Also critical is the ability to test patches prior to deployment, deploy patches in a systematic fashion, and then audit the working environment for compliance.

Back to Top

Back to Top

Back to Top

Back to Top

Back to Top

Back to top