Kubernetes tools are helpful for your team and sadly, your attacker

Overview

Defenders must constantly stay aware of the latest attacker trends to ensure their organizations’ assets are protected. In recent years, leveraging commonly found binaries on Windows/Linux systems has become more popular with offensive security professionals. The methodology of “living off the land,” and using a system’s native binaries is attractive to adversaries and red teamers alike. After all, why would one increase their forensic footprint by introducing a new tool when a comparable option is shipped by default on popular operating systems? This tradecraft methodology has spawned numerous blog posts, tweets, and notably two Github repositories([0][1]) tracking numerous ways to achieve various steps of the attacker’s kill chains without introducing new tools.

The previously mentioned Github repositories largely focus on traditional Windows/Linux binaries. With the growth of various DevOps tools around administrating Kubernetes, Lacework Labs has identified techniques that defenders need to be aware of to protect their respective environments. Lacework Labs previously documented how the lack of audit logging presents a unique channel for attackers to abuse. In this post, we further explore how kubectl could be abused in Kubernetes environments and what defenders need to know.

Payload Hosting and Data Exfiltration with Kubectl

Kubectl is the standard utility used to interact with the Kubernetes API to deploy and manage Kubernetes resources. An interesting kubectl feature is the proxy flag. Per the help page:

“Creates a proxy server or application-level gateway between localhost and the Kubernetes API server. It also allows serving static content over specified HTTP path. All incoming data enters through one port and gets forwarded to the remote Kubernetes API server port, except for the path matching the static content path.” — Kubectl help page

Let’s explore the ability to arbitrarily serve static content. By specifying a directory to serve along with listening on a specific (or all) interfaces, it is possible for kubectl to act as a HTTP server for payload hosting. Figure 1 below shows the code snippet to achieve this. Within the bottom half of the terminal, an attacker can query a rest endpoint and pipe to bash their payload script. This methodology could also be applied to exposing directories on a victim host for data exfiltration.

While the above example is akin to a web server, additional command line options (–accept-hosts) can be specified to limit hosts that can connect to the server. This creates a pseudo firewall only permitting specific hosts based on IP.

Pivoting via Kubectl Proxy

Exploring the proxy flag further, we’re effectively leveraging kubectl to listen on a specified interface and port to tunnel traffic to the Kubernetes API server. However, it’s also possible to extend this to point to any HTTP, or even TCP endpoint. By modifying the kubeconfig to add an additional “cluster,” one can arbitrarily set any IP/port combo they desire. This enables adversaries to tunnel traffic through kubectl to a desired internal endpoint, such as a metadata service in AWS EC2 instances.  This requires an adversary to have access to a compromised host where they would be able to modify an existing kubeconfig file or introduce their own. Figure 2 below shows the workflow described above.

While the above example shows how to abuse the proxying behavior to point to arbitrary endpoints, the port-forward flag can be used to interact with specific pods within a Kubernetes deployment. This could be advantageous to those with a large Kubernetes presence hosting critical business applications. Ensuring that appropriate audit logging is occurring within your Kubernetes cluster can help quickly identify potential abuse.

Building Triggerable Events with Kubectl

Finally, a triggerable event (think logic bombs) can be achieved with kubectl’s “wait” command. This functionality can be used to trigger a script or binary to execute after a specific Kubernetes resource has been deleted, created, modified, etc. By default, the timeout value is set to 30 seconds, but can be modified with the –timeout flag. If the event occurs within the timeframe, then the following bash command after “&&” will be executed. If the event has not occurred (such as a pod being deleted in the example below), then the follow-on bash command will not be executed.

$> kubectl wait --for=delete pod/tmppod \
        --timeout=300h && echo 'do evil thing here'

After setting up a kubectl wait command, the pod can be deleted ($> kubectl delete pod/tmppod) to trigger the echo bash command. The given example would execute only once, but further tooling with system timers or cronjobs can build triggerable events around Kubernetes resources.

Conclusion

Adversaries will continue to develop tradecraft against the ever-growing cloud/DevOps landscape. Understanding how these utilities are normally used, and how they can be abused is critical for enterprise defenders to stay ahead of adversaries and hunt for suspicious activity. To see more content like this, follow Lacework Labs on LinkedIn, Twitter, and Youtube and stay up to date on our latest research!