Kubernetes has allowed us to shift from a server-centric deployment mindset to an application-centric deployment mindset. This sometimes makes us forget that it’s all just the orchestration of workloads on servers.
Many Security and Operations teams spend considerable time and effort securing that orchestration layer by securing Kubernetes workloads and cluster configurations. In doing so, they sometimes overlook the security of those underlying servers.
What’s the risk?
There are many layers to the Kubernetes infrastructure and it’s important to ensure the security of each one. Even if you achieve perfect security at one layer, the cluster is still vulnerable to attack if other layers are lax.
The cluster nodes are critical components of the cluster to secure. This is because they contain your organization’s running applications and credentials to communicate with the greater cluster. Common security misconfigurations or simple outdated packages with CVEs allow attackers to compromise the cluster nodes. Which can result in business disruption or compromised secrets/customer data.
CVEs, CVEs, CVEs
Critical components on your Kubernetes cluster nodes require constant attention to reduce security risk. A major risk factor is CVEs present in outdated packages. Deploying clusters with fully updated nodes is not enough. Those systems quickly accumulate critical CVEs, allowing attackers to gain access to your nodes.
Over the last 20 years we’ve seen rapid growth in the number of reported CVEs. With the trend accelerating in the last 5 years.
*2022 total is estimated based on the current monthly trend
Several of the components found on even the most bare-bones Kubernetes cluster nodes have still seen a large number of critical CVEs over the last two years:
- 6 containerd CVEs
- 2 Docker CVEs
- 2 Linux kernel cgroups CVEs
These CVEs are key links in the kill chain of many modern Kubernetes breaches. Attackers compromise an external application running on the cluster. Afterwards, they then exploit vulnerabilities in the runtime or Linux cgroups to escape the application container.
From there they escalate their privileges using additional vulnerabilities and gain access to the larger cluster or business infrastructure. Patching the vulnerabilities that attackers use to escape the application images breaks the kill chain. Which in turn eliminates many of the common cluster attack methods.
Don’t forget the obvious
Not every link in a kill chain is as sophisticated as a cgroups container escape. Sometimes it all just comes down to plain old misconfigurations. These are a particular problem on Kubernetes cluster nodes. There are numerous places within Linux distros where services can be misconfigured to:
- Accept insecure protocols
- Skip authentication
- Expose sensitive information on the system or services
An excellent example of a potentially misconfigured service is OpenSSH, which runs on most servers. OpenSSH could be a potential link in a kill chain when another system in your environment has already been compromised.
The default configuration of OpenSSH is fairly insecure. Most distributions include some of these common settings that attackers can use to gain access to your systems:
- Root logins permitted
- High numbers of login attempts allowed with a low backoff time
- Password authentication permitted
Even worse, once attackers breach systems, the out-of-the-box configurations lack auditing and often have lax permissions. Thus allowing attackers to gain further access. Lack of auditing is particularly troublesome because it makes it impossible to determine how attackers compromised the system. It also makes it hard to examine what extent your infrastructure was compromised.
Using Mondoo to secure cluster nodes
At Mondoo, we believe now more than ever that security requires a layered approach. Our full-stack Kubernetes security solution addresses security at each layer of Kubernetes infrastructure. From the container image to the cloud it runs on, Mondoo surfaces security concerns early in the development process where they are the easiest to remediate.
Mondoo detects security misconfigurations as well as vulnerable packages in all major Linux distros (Windows/macOS too). Out-of-the-box policies help you harden your node’s OS and kubelet configs and CIS/BSI policies make compliance a breeze.
