Prioritizing your container image pipeline
By: Julien Sobrier, Manager, Product Management
September 19, 2022
In a previous blog post, we shared best practices and tips about building a container image pipeline to prevent vulnerabilities from reaching production. But many users of containerized applications have to deal with a number of vulnerabilities that already exist in active containers running in production. It can be challenging to decide what vulnerabilities to fix first, what criteria to use to prioritize the work items, and what images to upgrade and in which order. Let’s look at a few indicators that your organization can use to help remediate the most impactful vulnerabilities.
Which ones of the 1,000+ vulnerabilities should you fix first?
Every public vulnerability gets scored from 0 (lowest severity) to 10 (highest severity) using the Common Vulnerability Scoring System (CVSS). There are several versions of the CVSS framework, but we’ll talk about the 3.0 version here.
Each CVSS score reflects the three aspects of a vulnerability:
- The vulnerability itself: its exploitability (for example, attack vector, attack complexity, and scope), and the impact on confidentiality, integrity and availability of a system. These parameters do not change over time.
- The activities around its exploitation: the quality of exploit codes, the ease of remediation and the probability of exploitations. These parameters can change over time if the vulnerability gets exploited in the wild and if breaches are reported.
- The customer environment in which the vulnerability is present: how much the confidentiality, integrity and availability of the affected system matters to the user.
These parameters should be set by the user. For example, a test environment would have a lower requirement for availability and integrity, which might lower the overall CVSS score as compared to the same vulnerability present in a production environment.
In practice, very few tools and users are able to update the CVSS parameters over time or to take into account the requirements of a specific environment. But the parameters remain a good approximation of the relative importance of vulnerabilities. CVSS is used widely, often as the main criteria for companies to rank vulnerabilities and prioritize which ones to fix first. As an example, the Log4j vulnerability has a CVSS score of 9.8 out of 10.
Vulnerabilities can be rated as low, medium, high, or critical. But the severity can come from two different sources:
- The severity associated with the CVSS score: a score of 0.1 to 3.9 is considered low, 4.0 to 6.9 is medium, 7.0 to 8.9 is high, and 9.0 to 10 is critical
- The severity given by the OS vendor or library vendor
You might think that a vulnerability with a critical CVSS score would be rated as critical by Redhat or Canonical (Ubuntu) when present in their OS. But this is not the case. In the majority of cases, the vendor severity is completely different from the CVSS severity. Here is an example mapping the severity reported by Redhat compared to the CVSS severity reported by the (National Vulnerability Database (NVD)), the main source of public vulnerability disclosure:
Out of 90 vulnerabilities reported to have a High CVSS severity, Redhat reclassified:
- 23 vulnerabilities as Critical
- 24 vulnerabilities as High ( same as CVSS)
- 35 vulnerabilities as Medium
- 8 of vulnerabilities as Low
Out of 152 vulnerabilities reported to have a Low CVSS severity, Redhat reclassified:
- 7 vulnerabilities as Critical
- 32 vulnerabilities as High
- 62 vulnerabilities as Medium
- 51 vulnerabilities as Low (same as CVSS)
This is quite a difference! The vendor severity should better represent the exploitability and impact of a vulnerability, taking into account the default configuration and remediations that might be present in the OS, etc.
The same vulnerability may have different severities on a different OS, which may differ from the severity listed by NVD. If you have more trust in your OS vendor to rate vulnerabilities for their own software, you might want to use their score in place of the CVSS score.
Companies who have to adhere to a compliance framework such as PCI DSS need to show that they have a process to remediate vulnerabilities in a timely manner. Redhat has created the Container Health Index, which grades container images, rather than individual vulnerabilities, based on the severity of vulnerabilities present in an image and their time of exposure. For example, grade B requires all critical vulnerabilities to be addressed within seven days and all high severities to be fixed within 30 days.
Users who can track how long vulnerable images have been running — by using Lacework vulnerability management capabilities, for example — can set similar internal policies to ensure all vulnerabilities are addressed within a certain timeframe they are comfortable with. This is also a good way to measure how effective a company is at updating images in production, independently of how many new vulnerabilities may have been found in a given week or month.
Other useful criteria includes how many images are affected by a vulnerability. You may want to fix a high severity vulnerability present on hundreds of running containers before fixing a critical vulnerability that shows up in just a couple of containers.
If a vulnerability is found in the base image, it might be very easy to update that base layer across multiple container images and decrease the total number of vulnerabilities in your environment quickly. Lacework shows you in which layer a vulnerability has been found, making it easy to figure out whether a base OS can simply be updated with a fixed version of the affected package. Assessing the impact of a vulnerability is a good way to decrease the amount of work and risk.
We mentioned that the CVSS framework includes information about the exploitability of a vulnerability: if it requires local access or is remotely executable, if it can be triggered automatically or needs user assistance, etc. A company with good visibility into their environment can map these parameters with the actual runtime configuration of the containers to help prioritize the most exploitable images.
For example, remotely exploitable vulnerabilities on containers exposed to the internet should be prioritized first. Local exploits on container images that don’t include a shell could have a lower severity. Vulnerabilities that can impact the underlying host might be prioritized on privileged containers or containers running as root.
This approach helps improve the security of a system, but it requires a way to correlate vulnerability information with container configurations. A comprehensive platform like the Polygraph® Data Platform, which includes vulnerability management, runtime visibility, and compliance, can correlate the runtime configuration with the relevant vulnerability factors to surface the “real” vulnerabilities that pose the greatest risk to your unique environment.
Complete risk posture
The holy grail is to use all of the criteria above (severity, length of exposure, impact, and exploitability) to come up with a comprehensive score that gives the most accurate ranking of vulnerabilities for your environment. Lacework has introduced a Vulnerability Risk Score that continuously monitors vulnerabilities and the containers on which they are running to give an up-to-date score from 0 to 10. This uses not only configuration information, such as network exposure, but also dynamic runtime information including network behavior, process activities, and alerts for each container.
Vulnerability Risk Score from Lacework
Companies can use any of these criteria depending on their level of maturity and visibility, and whether they need to focus on security or compliance. The prerequisite is the ability to get a full inventory of not just all images scanned and their vulnerabilities, but also a complete list of active images. Regardless of what criteria is used to prioritize what images to fix, it is important to be able to track the progress and measure how teams are doing against the objectives.
Stay tuned as Lacework adds advanced features,including new third-party vulnerability feeds to enrich the vulnerability risk scores, up-to-date details on exploits seen in the wild, additional attributes about the vulnerability, and a more granular list of affected versions across different platforms. This will bring our customers a more dynamic view of vulnerabilities to help them focus on the most critical fixes or mitigations. We will share more about these new developments in the coming months.