Patching Runc Vulnerability (CVE-2019-5736)
So far, most of our posts on this blog have been about the exciting parts of running our own Kubernetes cluster. But, running a Kubernetes cluster isn't all having fun deploying applications. We also have to be responsible for system maintenance. That need became particularly apparent recently with the release of CVE-2019-5736 on 2/11/19.
What is CVE-2019-5736
CVE-2019-5736 is the CVE for a container escape vulnerability discovered in runc. Runc is a CLI tool which runs containers in accordance with the Open Containers Initiative (OCI) spec.
As we said, this vulnerability is a container escape vulnerability. Essentially, a process running as root inside the container can exploit a bug in runc to gain root privileges on the host running the container. Once the malicious process has root privileges on the host, it can pretty much do whatever it wants.
This vulnerability is particularly concerning as it is relatively common to run public container images on a k8s cluster. Until this vulnerability is patched, we're placing high levels of trust in the creator of the public container image that they aren't doing anything malicious, and also trusting the container image isn't tampered with when pulled down to run on our k8s cluster.
This blog post from the k8s team offers even greater detail on runc and the vulnerability in question.
What is CVE-2019-5736's impact?
As users, it is rare that we would ever interact with runc directly. So, we may be tempted to think this vulnerability won't impact us. However, runc is a foundational component of Docker, as Docker uses runc to do the heavy lifting of Linux container interactions. Thus, anyone using Docker to run containers is vulnerable. Additionally, most k8s clusters, including ours, use Docker as the tool for actually running container images as containers. As a result, we want to be sure that we address this vulnerability both on our local systems where we interact with docker directly, and on our k8s cluster.
Addressing CVE-2019-5736
As mentioned in the previous section, we need to patch both our local Docker installs and our k8s cluster.
Patching local Docker installs
Patching the local Docker install will be operating system dependent. However, regardless of operating system, we need to ensure we are using a version of the Docker engine which address the vulnerability. Docker lists the versions addressing the vulnerability in this blog post.
We do our development on a Ubuntu-based distro, and install docker-ce via
Docker's repositories,
so our update process is the following:
$ sudo apt update
$ sudo apt install --only-upgrade docker-ce
When we run docker version, we should see a
Server.Engine.Version which is greater than or equal to the versions
addressing the vulnerability listed in the blog post.
Patching k8s cluster (w/ kops)
Our method for creating/managing our k8s cluster dictates our method for ensuring our cluster uses a version of Docker which addresses the vulnerability.
Our cluster was created using kops, so we will discuss how to use kops to ensure our cluster is protected from this vulnerability.
The kops advisory for CVE-2019-5736 discusses a number of mitigations. We decided the best one for us was upgrading to kops 1.11.1.
Before following the rest of this blog post, please check that both your current kops and k8s version support you updating to 1.11.1.
Kops 1.11.1 includes a number of mitigations for the CVE, but the most important component is kops configures our cluster to start using Docker 18.06.3, which addresses the CVE, instead of 17.03.2, which does not.
If, after installing kops 1.11.1, we run kops update cluster, we'll see the
LaunchConfiguration for our hosts now specifies the new Docker version. We
then run kops update cluster --yes to actually apply these changes.
These changes will require a rolling update to actually take effect, so we want
to run kops rolling-update cluster to preview the change, and kops rolling-update cluster --yes to enact the change.
After doing so, we should be able to ssh into our master, run docker version
and see we are using a supported version, as we specify below:
$ ssh admin@k8s-master-ip
$ sudo docker version # should be 18.06.3
Thanks to the upgraded Docker, our k8s cluster is no longer vulnerable to CVE-2019-5736. As mentioned above, the [kops advisory for CVE-2019-5736] (https://github.com/kubernetes/kops/blob/master/docs/advisories/cve_2019_5736.md) provides a number of different mitigations, so upgrading Docker via upgrading kops to 1.11.1 is not the only way to address this CVE.
Preventing these types of problems
As mentioned in the explanation section, this vulnerability could only be exploited by processes running as root within the container. Fortunately, it is very rarely a good idea for a process to run as root within the container, so another way to address this vulnerability is to ensure all the containers we run on our k8s cluster conform to the best practice of running as non-root. This [blog post] (https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/#8-run-containers-as-a-non-root-user) from the k8s team gives more instruction on how to enforce containers running as non-root at a cluster wide level. The majority of popular images do not run as root by default, so this guideline should not be particularly difficult to enforce.
Additionally, this vulnerability serves as a good reminder to only run images that we trust on our Kubernetes cluster.
While we should still patch our clusters regardless, vulnerabilities such as CVE-2019-5736 serve as a good reminder that following best practices can have important preventive security ramifications.
Conclusion
Hooray, both our local Docker and k8s clusters are secure against CVE-2019-5736! A huge thanks to the members of the kops, k8s, Docker, and runc communities for their work on making patches available in a timely manner.