Review Conference Materials

February 25 – CNCF Hyderabad – Koti Vellanki

May 19-21 – Kubernetes Community Days Czech & Slovak 2023 – Koti Vellanki

May 26 – DevOpsPro Europe – Jacob Beasley

June 5th – KCD Columbia – Koti Vellanki

June 15 – DevOpsCon Berlin – Jacob Beasley

September 28 – DevOpsCon New York – Jason Woodman

September 28-29 – Hack in Paris – Jacob Beasley

Going forward, we will be doing, at a minimum, a quarterly security update in which we will upgrade to the latest version of each open source tool as well as run automated tests to validate all tools are working properly with as many Kubernetes versions as possible. As of this posting, our tests are run against versions 1.16 through 1.26.

• Project Falco performance is improved
• Support for setting minimum priority in project falco (to reduce chattiness)
• Using Project Falco Sidekick app to forward events to Dash
• Support for generating service tokens with newer versions of kubernetes
• Google Container Registry support is finished
• NestJS, Kube-Bench, Kube-Hunter, Trivy, Kubesec, and Falco upgraded to newer versions
• Automated tests for Gatekeeper completed
• Various UI Fixes

Feel free to use this as-is or to download and modify it to your hearts content!

Slides

Lab Guide

After seeing the dire need for Kubernetes security tooling, Intelletive Consulting makes Kubernetes security accessible for all organizations. Read more.

Project Falco Anomaly Detection

Project Falco can stream suspicious events to m9sweeper. Now you can tell Project Falco to notify you about any new events that it sees. This is very powerful and provides you security team with the ability to triage issues discovered by Falco in real time.

Improved Falco Event Details Page

Now you can see other times this same event has occurred as well as the number of times per day it is occurring.

Override CVE Severity

Sometimes your security team may identify that a particular CVE is actually more severe than the Trivy scanner’s database indicates. Now you can create an override type exception to override the CVE’s severity when it is found during a scan.

We found ourselves having to build Dockerfiles and helm charts from scratch for every one of our customers. With this project, we will never have to do that again, and we hope to make it easier than ever for others to do the same!

Version 1.2.1 includes a number of new features and technical changes:

1. Project Falco has now been integrated into M9sweeper.
2. Licensing has been removed. Now all features are available regardless of licensing. The licensing portal still is available for those who wish to monitor that m9sweeper is still running in each of their clusters, but it is purely a monitoring/metrics aggregation tool now (and this may be expanded in the future for those who wish to run it as a monitoring tool on-premise).
3. Many new reports have been added.
4. It is now structured as mono-repo to make open source contributions simpler.
5. All Components are now versioned together. This simplifies deployments/releases considerably.
6. A documentation site has been released with instructions for installation.
7. Configuring docker registries during installation has been greatly simplified.
8. Support for Elastic Container Registry using AWS Secret Keys has been integrated into Trawler.
9. Support for webhooks with Azure Kubernetes Services has been added (there is a wiki with instructions in the documentation site).

Kubernetes is an extensible system, but also a very complex one. In enterprises, it is common to define a set of best practices about how to use Kubernetes as well as how to build software, and then to attempt to get everybody doing things the same way. 

If you have ever tried to get multiple different teams to do things the same way, you know that that is hard. Once things are in production, nobody ever wants to change things because they are already focused on the next feature or next milestone. 

So, what if there was a way to define the best practices (with code or otherwise) and then to report on how compliant teams are with your best practices? And what if you could even prevent things from being deployed that did not follow best practices? 

That is essentially what Gatekeeper does. You can define your requirements using constraints, which are a combination of configuration details (constraints) as well as corresponding code in a language called rego (constraint templates). Together, the rego code and the configuration parameters allow you to define, measure, and enforce compliance with Kubernetes. 

Gatekeeper Examples – Without M9sweeper

You can install Gatekeeper on a small cluster using a one-liner: 

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.7/deploy/gatekeeper.yaml


However, in an enterprise environment we recommend using their helm chart and tuning it so that it is easier to perform upgrades as well as tune the proper CPU/RAM based upon your needs. 

Once installed, you need to create a constraint template, such as this: 

And a constraint, tying that particular requirement to a specific set of namespaces: 

Learn more here

Gatekeeper Examples – With M9sweeper

With M9sweeper, you can install constraint templates from a library of prewritten constraint templates (based upon the official open source gatekeeper library): 

Then, you can create constraints for specific namespaces or pod: 

Conclusion

When you have multiple teams deploying software to Kubernetes, it is important that you make sure teams are doing so in a professional manner that prevents any one team from deploying software that might cause issue for another team at your company. 

Gatekeeper helps you to configure rules that developers must follow when deploying software, or it lets you assess how well the team is doing when deploying software. And, as always, M9sweeper makes deploying and managing these tools easy for you and your team. 

We will approach the topic by starting with how an attacker might gain access to an environment and then how they might break out to cause more damage. This is not an exhaustive list – it is just a basic list of some of the simplest ways to break into a Kubernetes cluster as well as instructions for preventing these kinds of attacks. 

Method 1: Break into Kubernetes Directly

There are a few ways you might be able to break into Kubernetes directly: 

1. Exploit known vulnerabilities in the operating system or libraries your software uses, such as known vulnerabilities in an XML Parsing library or a certain version of OpenSSL with Apache Web Server. 
2. Connect to the Kubernetes API: If Kubernetes does not have Role Based Access Control enabled or has enabled authentication methods that are insecure, you may be able to use the Kubernetes API to deploy and run containers+scripts in the Kubernetes cluster. 
3. Steal Someone’s Authentication Credentials: If unencrypted API traffic is allowed, you may be able to packet sniff and steal authentication credentials from someone else who is using the Kubernetes’ APIs and then deploy arbitrary code. If developers have been granted too many privileges and their credentials are stolen (through various means – the most common is accidentally committing them to github or a phishing scheme), you may be able to connect to the Kubernetes API. 

Once a hacker has gained enough access to the Kubernetes APIs, the exploitation is simple – they just list what is running, deploy their own containers/code (ideally with escalated privileges), and modify what is running to inject sidecars/listeners to cause as much harm as possible. 

Prevention Methods

1. Follow Kubernetes Best Practices: Run a tool like kube-bench (note: m9sweeper can walk you through this) to ensure your Kubernetes environment is properly setup with non-encrypted ports being blocked, role based access control enabled, etc. 
2. Ensure regular virus scanning and updates to your nodes (or better yet, use an operating system like Core OS or CentOS Stream that comes with next to nothing in it and is regularly patched)
3. Firewalls: Require someone to be within a VPN or to tunnel through a jumphost in order to be able to connect to the Kubernetes APIs. 
4. Ensure regular scanning of your code and its underlying libraries, such as using sonarcube to scan your code and trivy to scan for known vulnerabilities (you can bet a hacker would do the same if they gained full or partial access to your systems). 
5. Principle of Least Privileges: Use firewalls to lock down all ports except those which you intend to be open to the internet. This prevents random operating system services you are not using from being exploited. 
6. Make it hard for attackers to know where to look: Block some HTTP headers and other kinds of indicators that would show what frameworks, libraries, and software versions you are running under the hood to make it harder or hackers to determine how they might break into your application. 

Detection Methods

Tools such as Falco can actively monitor Kubernetes API usage for suspicious activity, alerting you if someone has gained access and is in the process of iterating through each of your applications in order to attack the cluster. 

Tools such as Gatekeeper or pod security policies can prevent users who have gained access from deploying code as root or otherwise with escalated privileges. 

Method 2: Break into Docker

Most Kubernetes clusters are running docker on every single one of their nodes. It is used to actually run the container code that kubernetes deploys. Later versions of Kubernetes have replaced docker with containerd (which is more secure and lighter weight), but earlier versions of Kubernetes are likely to still be using Docker. 

Docker, under the hood, just calls containerd to run your code in containers, so why was docker even needed and what does it do? Docker is an API and set of Command Line tools for building, executing, and managing currently-running containers. When it is running on a machine, it actually runs a web API that usually has no security and, in some cases, is even exposed on the internet! This allows an attacker to connect from outside or inside the cluster and instruct docker to run arbitrary code, such as booting up a container running as root and executing a script. From there, they can perform a container breakout attack to gain further access. 

Prevention Methods

Later versions of Kubernetes do not use Docker anymore. This offers many benefits beyond just security – docker is also bloated and proprietary (it will soon cost money). Docker is still an excellent tool to compile and build your docker images, but it is not the best way to run your docker images in a Kubernetes cluster.

For this reason, we recommend upgrading and ceasing the use of docker within your Kubernetes clusters. 

If that is not an option or has not been implemented, be sure the ports Docker’s APIs run on are firewalled off such that no outside actor can connect to them. 

Detection Methods

Just do not use Docker in a Kubernetes environment anymore. You could track docker’s logs for issues, but the industry at large is simply moving away from running Docker in Kubernetes, so we recommend focusing your efforts there. 

Method 3: Container Breakout

Once you have established the ability to deploy code, the next step is to break out of the container into the linux host and, from there, utilize traditional hacking techniques to systematically spider out into the entire organization. 

Container breakout is just what it sounds like – it is when a container process breaks out of its container and gains full access to the host operating system. The easiest way to do this is to deploy a container running as root and then use the chroot command to change the container’s root folder from the container’s own folder to the root linux operating system’s folder. From there, there are any number of ways to open the machine up to an outside attacker. 

Prevention

Pod Security Policies/Gatekeeper: The absolute simplest way to prevent container breakout is to enable Pod Security Policies that prevent running as root, disable escalating privileges through sticky bits, and disable writing to disk (note: M9sweeper ships with Gatekeeper policies that have all the same features as PSPs but work with newer versions of Kubernetes). Coupled with regular operating system updates and security scans, and you have yourself a pretty solid strategy to prevent container breakout. 

In addition to this, further tuning of SE Linux/App Armor can be used to further secure the system, and regular updates of the linux kernal/operating system are necessary in case any security exploits become available. 

In extreme situations, you might look at kata containers, a method in which a separate virtual machine is spun up for each container. They are heavier than containerd, but they essentially make container breakout impossible because even if you break into the kata virtual machine, it is heavily locked down and you have not actually broken into the host node’s operating system in any way. Even so, kata containers add additional complexity, so in most cases it is not needed. 

Conclusion

We have quickly given you an overview of how Kubernetes could be broken into by potential hackers. If you know much about security, you know that most of these issues are not unique to Kubernetes, but are really present anywhere you deploy software. Kubernetes (as with anything) should be configured properly to make it more secure and more difficult for attackers to attempt to exploit. 

Stay tuned as we introduce a few tools that your team should be using to secure your cluster and make these kinds of attacks extremely difficult! 

This can serve as a template for an internal security assessment that your team could perform (based largely on the other articles we have sent out), or you could hire us to perform this assessment. We have experienced, certified Kubernetes Security experts on staff and would be happy to perform this assessment.  

Facets Covered: 

The Linux Foundation likes to divide Kubernetes Security into 3 facts: Cloud, Cluster, and Container Security. 

Cloud Security

This facet is actually, ironically, not very much about Kubernetes, but it is important to call out. The underlying infrastructure and the cloud accounts (VMWare, AWS, Azure, Google, etc) used to manage that infrastructure must be secured, access limited, and devices kept up to date with the latest security patches. 

I will, nonetheless, mention several things here: 

1. 2-Factor Authentication: Make sure all cloud accounts have 2FA so that remote hackers have a significantly harder time gaining access to an account. 
2. Virus Scanners and Security Patches: Make sure your physical devices and your employees’ physical devices have up to date security patches and regular virus scans to detect if their devices have been compromised.
3. Virtual Private Networks: Make it so that someone has to authenticate with a VPN from their device to access your cloud accounts or to connect to your Kubernetes Security cluster or, in smaller environments, at least use IP-address whitelisting to prevent unauthorized connections. This significantly reduces your attack surface. 
4. Web Application Firewalls: Use WAFs to block many malformed and/or suspicious HTTP requests. Most cloud providers now offer pay-as-you-go access to WAFs which can protect against many forms of buffer overflow or denial of service vulnerabilities. 
5. Billing Alerts / Monitoring: Add billing alerts so that if a hacker spins up new servers or otherwise incurs charges, you get alerts. 99% of hackers are script kiddies spinning up bitcoin miners.
6. Encryption: Enabling proper encryption on load balancers, wifi, email, and your VPN so that sniffers cannot pick up keys, tokens, or authentication tokens and gain unauthorized access to things. 
7. Access Audits: Regularly review and limit users’ access to just the systems they truly needs access to. 
8. Training: Regularly give your staff security awareness training to help them to identify suspicious requests from potential hackers.  

I will not spend much time on this as this facet of the 3 is covered by an established field of materials in the CISO/CISSP space. 

Cluster Security

You need to make sure that your cluster is secured. 

1. Cluster is Up To Date: Do not run extremely old versions of Kubernetes (those that are end of life) and ensure that you are running the latest security patches. 
2. CIS Benchmarks: Run tools like Kube Bench to assess that you are following best practices related to cluster security – everything from disable insecure modes of authentication (ex: anonymous authentication) to insecure encryption settings.  
3. Pen testing: Run tools like Kube Hunter regularly to perform benchmarks and detect regressions in your cluster’s configuration. 
4. Intrusion Detection: Tools like Project Falco can detect and alert on suspicious activities, such as command shells being opened in production. 
5. Network Policies: Setup default network policies cluster-wide that limit access between different namespaces. 
6. RBAC: Ensure that developers and teams are given limited privileges and just privileges to the applications that they are working on. 

Container Security

We should make sure that the applications you are deploying as well as the way you are deploying them is secure:

1. Trivy: Scan your applications for obvious, repairable vulnerabilities from the vulnerabilities database. 
2. KubeSec: Scan your pod settings for obvious shortcomings or improperly configured privileges. 
3. Gatekeeper / Pod Security Policies / OPA: Ensure and prevent applications from being deployed with escalated privileges. Run Gatekeeper in audit mode initially to identify problem and then begin enforcing after you have had time to modify your deployments to deploy securely to prevent regressions. 
4. Network Policies: Limit your applications to only be able to connection to applications they have a legitimate reason to connect with. 
5. Linux Privileges: Configure Security Contexts’ Linux Capabilities, AppArmor profiles, and/or Secomp profiles to ensure applications have the minimum privileges that they require. 

We hope you have enjoyed this series! Please reach out to us with feedback or questions, and let us know if there is any way we can help!