Kubestalk: Uncovering Hidden Security Risks in Your Kubernetes Clusters

Kubestalk is a powerful GitOps tool that simplifies Kubernetes management by integrating with Git repositories. By adopting a GitOps workflow, you can automate the deployment, updating, and management of Kubernetes resources based on changes in Git. This blog post will guide you through installing Kubestalk, understanding GitOps principles, and exploring real-world use cases with code examples.

Introduction to Kubestalk

Kubestalk is an open-source GitOps tool for Kubernetes, developed by the Kubestalk team. It allows you to manage Kubernetes resources using GitOps principles, which means that all configuration changes are made through code stored in Git. This approach provides a number of benefits, including version control, collaboration, and automated deployments.

Kubestalk works by watching a Git repository for changes to configuration files, and then applying those changes to a Kubernetes cluster. This allows you to use Git as the single source of truth for your Kubernetes resources, and makes it easy to roll back changes if necessary.

Mapping the Attack Surface

Working Architecture

First things first, let us get a quick understanding of the working architecture of Kubernetes.

Kubernetes Core Architecture

A Kubernetes cluster consists of a set of worker machines, called nodes that run containerized applications. A pod is the smallest unit in Kubernetes, hosting one or more containers. A control plane/master node manages the worker nodes and the pods in the cluster.

The control plane includes components such as:

Kube API Server: A frontend for Kubernetes that takes RESTful requests from clients and translates the requests to operations on containers, pods or nodes.
etcd: A key-value database used as a backing store for all cluster data.
Kube Scheduler: A watcher for newly created pods.
Kube Controller Manager: Runs controller processes.

A few components on each node maintain the running pods and provide a runtime environment to it:

Kubelet: Makes sure that the containers are running properly.
Kube Proxy: Maintains network rules on nodes.
Container runtime: Software that runs on containers (e.g. Docker).

Kubestalk Methodology

The initial idea was to cover the basic components of Kubernetes instances. However, the attack surface gradually widened during the course of research.

The scanning timeline spanned over a period of 59 hours distributed across many nodes spread across the internet.
Broken down our findings into two major parts.

The first part covers the core components of Kubernetes, and
Second one covers the additional attack surface which is external tools used to manage k8s clusters 🙁

The United States alone accounted for almost 57% of the total exposures, followed by Germany, Belgium, Ireland, Singapore and Japan. Findings showed around 61% of the exposures were hosted in public clouds, with Google Cloud Platform (GCP) leading the most exposure counts

Most affected versions – v1.21 and v1.22

An architectural overview of the tool is depicted in the flowchart below:

Additional Attack Surface – Kubernetes Management Components

Apart from the regular default components, we discovered developers using additional software to manage their Kubernetes infrastructure, which also poses a significant security risk if misconfigured
Let’s take a look at one component .

cAdvisor

cAdvisor is a container metric engine that provides developers with an overview of their resource usage and the performance of their containers.

The software exposes a Web UI for viewing the resource metrics. It includes system processes, CPU, memory, network and filesystem usage statistics.

Found a total of 23,101 unauthenticated cAdvisor dashboards exposed.

Deep Analysis of Compromised Clusters

The Deep Analysis of Compromised Clusters revealed the presence of adversaries deploying malicious cryptominers inside the compromised clusters.

It was discovered that many instances had traces and IoCs of the Hildegard malware. Furthermore, a script attributed to TeamTNT was found on one of the instances, which had a detection rate of zero on VirusTotal.

How script works

The script works by initially using masscan to discover unsecured Kubernetes clusters across IP ranges by scanning for open ports.
Once an unsecured cluster has been discovered, the script makes use of zgrab to probe for HTTP services serving the path /v1.16/version.
If an unsecured cluster that allows anonymous access to its APIs is discovered, launching a malicious process in a hijacked container is more stealthy as it doesn’t need to pull new images or run new containers.

A Hacker’s Point of View

As an attacker, gaining code execution (RCE) in a running container through unsecured API servers or Kubelets is possible.

Both the exec API and run APIs allow running arbitrary commands in containers.
Once an unsecured cluster that allows anonymous access to its APIs has been discovered, launching a malicious process in a hijacked container is more stealthy as it doesn’t need to pull new images or run new containers.
In case Kubelet’s APIs running on port 10250 allow anonymous access, an attacker can discover the pods running inside the node by visiting the /pods endpoint.
- One of the first steps would be to discover the pods running inside the node.
- A list of pods inside the container can be retrieved by visiting the /pods endpoint.

curl -sk 'https://<ip>:10250/pods' | jq -r '.items[] | "Pod: \(.metadata.name)\tNamespace: \(.metadata.namespace)\tContainer: \(.spec.containers[].name)"'

Installation of Kubestalk

KubeStalk is written in Python and requires the requests library.To install the tool, you can clone the repository to any directory:

git clone https://github.com/redhuntlabs/kubestalk

Once cloned, you need to install the requests library using python3 -m pip install requests or:

python3 -m pip install -r requirements.txt

Everything is setup and you can use the tool directly.

Command-line Arguments

A list of command line arguments supported by the tool can be displayed using the -h flag.

$ python3 kubestalk.py  -h

    +---------------------+
    |  K U B E S T A L K  |
    +---------------------+   v0.1

[!] KubeStalk by RedHunt Labs - A Modern Attack Surface (ASM) Management Company
[!] Author: 0xInfection (RHL Research Team)
[!] Continuously Track Your Attack Surface using https://redhuntlabs.com/nvadr.

usage: ./kubestalk.py <url(s)>/<cidr>

Required Arguments:
  urls                  List of hosts to scan

Optional Arguments:
  -o OUTPUT, --output OUTPUT
                        Output path to write the CSV file to
  -f SIG_FILE, --sig-dir SIG_FILE
                        Signature directory path to load
  -t TIMEOUT, --timeout TIMEOUT
                        HTTP timeout value in seconds
  -ua USER_AGENT, --user-agent USER_AGENT
                        User agent header to set in HTTP requests
  --concurrency CONCURRENCY
                        No. of hosts to process simultaneously
  --verify-ssl          Verify SSL certificates
  --version             Display the version of KubeStalk and exit.

Basic Usage

To use the tool, you can pass one or more hosts to the script.
A basic usage is as below:

$ python3 kubestalk.py https://███.██.██.███:10250

    +---------------------+
    |  K U B E S T A L K  |
    +---------------------+   v0.1

[!] KubeStalk by RedHunt Labs - A Modern Attack Surface (ASM) Management Company
[!] Author: 0xInfection (RHL Research Team)
[!] Continuously Track Your Attack Surface using https://redhuntlabs.com/nvadr.

[+] Loaded 10 signatures to scan.
[*] Processing host: https://███.██.██.██:10250
[!] Found potential issue on https://███.██.██.██:10250: Kubernetes Pod List Exposure
[*] Writing results to output file.
[+] Done.

Conclusion

Kubestalk is a powerful tool for identifying security vulnerabilities in Kubernetes clusters and the external tools used to manage them.

Its various modules enable security teams to assess different areas of a Kubernetes environment, including the nodes, containers, and network.

Additionally, Kubestalk can detect various types of threats, such as misconfigurations, privilege escalations, and malware.
By using Kubestalk, organizations can proactively identify and remediate security risks in their Kubernetes clusters, ultimately improving the overall security posture of their infrastructure.

It is important to note that Kubestalk should not be the only security tool used and that regular security assessments should be conducted to ensure the ongoing protection of Kubernetes environments.