System administrators that work in enterprise environments know that patching is practically a full-time job. Consider the effort involved in patching just one system: a sysadmin must determine that a patch is available, plan for downtime or disruption, download the patch, deploy the patch to the system, and ensure it recovers its previous state. 

It’s quite a process and that’s for just one machine. In the enterprise environment, there are hundreds of servers to manage, which means that the job of patching becomes an all-day responsibility. There’s also a significant risk of failed reboots once a patch is installed.

That’s why sysadmins need to think of patching from a patch management perspective. In our ultimate guide to Linux patch management, we explain how systems administrators can free up time and organize patches using automation tools, what sysadmins can do to better manage the risks around patching, and explain why live patching is a game-changing tool for enterprise patching.

Why Is Patch Management Different from Patching?

Administrators could simply patch Linux systems manually – going from machine to machine or node to node in order to patch. However, there’s a risk of human error, and it can be challenging to roll back the patch if there are any issues.

Where a patch goes wrong, it can lead to long downtimes, while patching manually can also be incredibly time consuming. 

Patch management benefits administrators by automating the entire process. Integrating a patch management system into your workflow will mean that you automatically detect updates, download them, and then deploy them to all servers. 

Sysadmins can take automation a step further by deploying live patching, which eliminates the reboot process that’s usually required after updating Linux. We discuss live patching in this section.

Why is Patch Management Important?

Unpatched public-facing web servers are a critical issue for cybersecurity, but cybersecurity isn’t the only reason to patch Linux. Patching also remediates bugs and adds new functionality. Large updates can add significant functionality to an operating system and may be required to maintain application compatibility in the long run.

When patches are not applied, they pile up. The longer administrators wait to patch a system, the more patching activity will be needed to get the system up to date. It increases the time it takes to fully patch a Linux server and compounds the risk that something is going to go wrong. 

Hotfixes available from vendors and distro developers are the most important patches to apply and should be applied immediately. Hotfixes address critical issues within the operating system and are given priority for good reason.

How Often Should Patch Management Be Performed?

Patching isn’t an occasional task. Patch management requires non-stop, continuous activity from sysadmins on almost every day of the year. Nonetheless, some patches must be applied immediately – while other patches can undergo a more extensive process of testing before deploying the patch.

When a critical security patch is released, it’s important that administrators test and deploy the patch as soon as possible. 

Zero-day vulnerabilities are a real threat to organizations and their digital assets. When a zero-day vulnerability is announced, threat actors quickly create exploits to take advantage of any unpatched systems. Unpatched vulnerabilities are a common way for attackers to find a way into enterprise systems – and are the most common attack vectors used by ransomware groups.

To lower the risk of a data breach, organizations should rapidly deploy security patches as soon as they are released.

For feature updates and less critical security fixes, patching should be done after thorough testing in a staging environment. Staging environments should be a replica of production to ensure that it’s a 1:1 match during testing, or errors could cause downtime in production. Even though testing is important, a good rule of thumb is to apply patches within 30 days of vendors making them available.

In a large enterprise environment, new updates are often released daily and that means constant testing and deployment. Manually checking for new patches every day is tedious, and that’s where patch management automation steps in.

Common Problems You Can Encounter When Patching

Patching is a critical activity, but it carries a few pitfalls. It’s worth keeping some of the common patch management pitfalls in mind when you build your Linux patch management strategy (which we’ll discuss in the next section):

Disruption caused by reboots: if you don’t use live patching, you need to be aware of the effects that reboots will have on your operations. This means scheduling maintenance windows or, in the case of high availability, ensuring that you only patch during off-peak times when your high availability system is not already overstretched. 

Patched machines won’t return to their previous state: similarly, by rebooting the machine, you run the risk that it won’t always simply start up and continue where it left off, and you need to be prepared to nurse it back into its previous state.

Running out of staff resources: even with the best of plans, you may still find that the sheer volume of updates and patches can become overwhelming. Prioritizing and making use of automation as far as possible is key.

Big changes to software: be mindful of large updates that can break existing functionality. Even where an update has been tested thoroughly, caution is still advised: the larger the update, the more attention and care you should pay when you distribute the update across the enterprise.

Updates may be buggy: with the exception of critical security patches, you should always test updates before deployment – but, even after rigorous testing, there is a risk that an update may turn out to be buggy in the long run.

Patches impact resource usage: in some cases, a patch can add so many new features and change the way in which a system works that the system ends up consuming far more resources for everyday tasks. This may not be revealed during testing and cause a slow-down in your operations.

To avoid some of the common problems you can encounter when patching, you should build a Linux patch management strategy and follow patch management best practices.

What are Linux Patch Management Strategies?

Patching automation is a key tool but, before jumping in and starting to patch, it’s worth talking about patch management strategy. 

Unlike closed-source operating systems like Windows, Linux patching can be a bit more unpredictable and complex. Open source has its advantages, but one disadvantage is running an operating system where changes are made by various contributors. Just one incompatible change could affect your entire organization.

To alleviate some of the overhead and hassles of poor patch management, here are a few strategies and best practices to incorporate into your procedures:

Create a patch management policy: This policy should include every step, including quality assurance (QA) testing, frequency of patching, any rollback procedures, and who it is that signs off on operating system changes.

Use scanning tools to find vulnerabilities: Whether it’s public-facing servers or internal hosts running corporate applications, vulnerability scans will find unpatched systems and help avoid common exploits.

Use reporting to identify failed patches: How do you know that a patch installed successfully? A good patch management solution offers a central dashboard that displays reports on successful and failed patch installations so that administrators can review and manually apply a patch if necessary.

Deploy patches as soon as testing is complete: Testing is important before deployment, but as soon as testing provides a green light for deployment, patches should be installed across the entire environment.

Document changes to the environment: Usually, documentation is done in the form of change control where authorized employees sign off on updates to the environment. This step is important when reviewing downtime and performing root cause analysis. It’s also important for auditing and compliance reasons.

The above list will give you a head start on your patch management strategy. In applying that strategy, you should also consider patching best practices.

Patching Best Practices

The SysAdmin, Audit, Network, and Security (SANS) organization is a good source for best practices for patch management. SANS best practice guidelines give administrators a roadmap on how to implement a corporate policy that documents, audits, and assesses risk across the organization to determine when and how patches should be deployed. The eight SANS best practice recommendations are:

Inventory your environment: To patch comprehensively, you know what needs to be patched, so you should build an audited list of all Linux systems on the network.

Assign risk levels to each server: Risk levels tell administrators which servers are most important and should be prioritized. All systems should be patched, but targeting the most important servers will lower the risk of them being compromised while testing and other patching tasks are underway.

Consolidate patch management software into one solution: Automation tools are beneficial, but too many different tools making changes to the environment can lead to errors and possible race conditions.

Review vendor patch announcements regularly: Automation tools will download updates automatically, but administrators should still be staying aware to understand when new patches are available, especially critical ones.

Mitigate risks of patch failures: It’s not uncommon for administrators to halt updates due to an issue with exceptions. When this happens, servers should be locked down, if possible, to limit exploit potential.

Always test patches in staging first: A staging environment should replicate production so that patches can be tested and lower the risk of downtime.

Patch systems as quickly as possible: The longer a server remains unpatched, the bigger the risk of compromise due to a known vulnerability.

Use automation tools: Automation tools take a lot of the overhead from administrators and automatically deploy patches when they are available.

Related read: Enabling Compliance with Faster Patch Management 

How Does Automated Linux Patch Management Software Work?

Automated patch management operates on several layers, and vulnerability scanning is one. To avoid becoming the next newsworthy data breach, organizations must do vulnerability scans on every device. 

Vulnerability scans identify if patches are missing, so administrators can deploy them as soon as possible. There are a few good vulnerability scanners available that make this first step much more efficient and convenient. These scanners are:

• Qualys
• Nessus
• Rapid7

With a scan complete, it’s time for patch management tools to take over. Several tools on the market make patching much more convenient for administrators. 

Better tools report on successful and failed patches so that administrators know when manual updates are also necessary. Patch management tools, therefore, deliver a comprehensive update on the current cybersecurity health of the enterprise environment. A few tools available to manage patches include:

• Yum – used in Red Hat Enterprise Linux (RHEL)
• Ansible
• Puppet
• SaltStack
• Chef
• Spacewalk

The above tools’ primary advantage is improved organization, because they download updates and then report the results to administrators. In that process, the right tool can minimize the disorganization that can occur when patch management is handled across a large environment. 

Administrators can also schedule patches, choose their own deployment policies, test, and then approve updates before deployment.

How Does Live Patching Fit into a Patch Management Framework?

Patch management tools offer enhanced organization to administrators, but reboots are still a major issue. Rebooting a critical Linux server means machine downtime for the organization and that downtime needs to be handled somehow. 

This could include high availability (though patching would still affect performance) or through scheduling maintenance windows, and scheduling patching during off-peak hours. This means that patches could be postponed until it’s convenient, which leaves unpatched servers vulnerable for longer. Live patching improves the entire process by eliminating the reboot process.

Eliminating reboots helps in a few other ways too – including in reducing the risks inherent to rebooting. What if the system doesn’t restart? And what if there are several critical servers that must be patched and rebooted simultaneously?

A sysadmin could potentially have several critical servers that power the entire organization that all need urgent patching for a vulnerability, and there is risk that several critical servers fail to restart smoothly. With live patching, that risk is eliminated.

KernelCare is a Linux live patching tool that integrates into existing patch management solutions. Patching is still scheduled, tested, downloaded, and deployed from the patch management tool. But KernelCare adds live patching capabilities and eliminates reboot requirements.

KernelCare live patching works in four simple steps:

1. Allocate kernel memory and load new, secure code into memory.
2. Temporarily freeze all processes in safe mode.
3. Modify functions and jump to new secure code, which plugs the vulnerability.
4. Unfreezes processes and resumes activity.

The magic is that you don’t need to restart the machine that is being patched to apply the patch. With KernelCare live patching, updated code is applied seamlessly with no sysadmin intervention and without any disruption.

If you’re using any of the patching tools I mentioned earlier (e.g., Ansible, Puppet, Chef, SaltStack) then you can use just the same tools to deploy KernelCare – you don’t need to install KernelCare manually on each server. With these tools, administrators can:

1. Distribute the KernelCare agent package (only necessary where servers have no internet access)
2. Distribute the KernelCare agent configuration file /etc/sysconfig/kcare/kcare.conf
3. Set environment variables
4. Install the KernelCare agent from either local or remote download servers
5. Register KernelCare with key-based or IP-based licenses

In addition to easy distribution and integration into current patch deployment applications, KernelCare also sends a “safe kernel” report to any vulnerability scanners that poll your servers for vulnerabilities. 

With KernelCare, your Linux servers are automatically patched without the need for reboots, and vulnerability scanners will report that your servers are updated and current with vulnerability patches.

How to Patch Your Linux Systems Manually?

Even when patch automation and live patching is in place, manual updates are occasionally necessary. For example, after a failed update, administrators may need to manually patch the system. Manual updates might also be necessary in a testing environment. The commands to update Linux depend on your distribution, but here are the commands for some common distributions.

For Debian-based distributions (e.g. Debian, Ubuntu, Mint), the following commands will display available patches and update packages and the operating system:

sudo apt-get update

sudo apt-get upgrade 

sudo apt-get dist-upgrade

For Red Hat Linux distributions (e.g. RedHat, CentOS, Oracle), the following commands check for updates and patch the system:

yum check-update

yum update

For SUSE-based Linux (e.g. Suse Linux Enterprise, OpenSuse), the following commands check for updates and patch the system:

zypper check-update

zypper update

Handling End-of-Life Linux

There’s an elephant in some server rooms. It’s called unsupported, or end-of-life, operating systems, and it’s something that can quickly trip up even the best patch management strategies.

An unsupported operating system no longer gets patches for newly discovered security vulnerabilities. With no patch, there’s no way to protect systems against critical vulnerabilities – even where automated patching is in place. Developing a custom patch is a difficult, expensive solution.

Why do enterprises run end-of-life operating systems? It can come down to pure neglect by the sysadmin team, where the needed action to upgrade to a supported version was never taken. More often than not, however, it’s a practical matter – like a specific piece of software that simply won’t work on the newer OS, or severe pressure on the sysadmin team that meant the upgrade couldn’t take place.

Thankfully, there’s a way to apply your patch management strategies to end of life operating systems too. You could subscribe to an extended support plan from the Linux vendor, though these can be expensive and include unneeded features. 

TuxCare’s range of Extended Lifecycle Support services for Linux distributions that are no longer officially supported is a more affordable alternative. It provides up to 4 years of security updates for a number of unsupported Linux distributions, as well as coding languages like PHP and Python.

Conclusion

Consistent patching is critical, but hard to get right. Reboots get in the way, and limited resources can let patching efforts down. Linux patch management is nonetheless at the core of keeping enterprise computing environments safe. 

Getting patch management right requires a strategic approach that follows patch management best practice – but that also makes use of the best tools out there to accomplish the task.

Integrating KernelCare into patch management reduces risk, improves the security of your Linux servers, and provides convenience to administrators. KernelCare also seamlessly works with your current patching process to introduce rebootless updates.

We have customers with servers that have not been rebooted in over six years. Thanks to KernelCare, over 300,000 supported servers in major data centers across the world keep their SOC 2 compliance status with our live patching framework. 

Try KernelCare now to help you keep to your Linux patch management strategy as well as take a lot of overhead and time-consuming processes off your administrators’ shoulders.

The post The Ultimate Guide to Linux Patch Management appeared first on TuxCare.

*** This is a Security Bloggers Network syndicated blog from TuxCare authored by Joao Correia. Read the original post at: https://tuxcare.com/blog/the-ultimate-guide-to-linux-patch-management/