Container Orchestration Demands the Right Security Approach

Advances in container orchestration, service meshing, and microservices have improved the lives of developers who are increasingly evaluated on their ability to deliver rapidly and continuously. With the support of containers, applications are more portable and can be deployed and scaled more quickly and reliably than in traditional deployment models. Automated continuous integration and delivery pipelines help ensure code is appropriately tested and staged before being moved into production. Errors in applications can often be quickly isolated to a specific container, fixed, and replaced with a new container running updated code.  

All these advantages have made containers an incredibly popular and useful way to build and deliver applications. They also create a completely new set of security challenges.

To understand how security is applied in, and for, containers, we should first understand what a container is and what it does. Simply stated, a container is software that packages code (and all the code dependencies) so the application runs quickly and reliably from one computing environment.  The container image becomes a container at runtime. But inherent container security simply isn’t adequate because of how the management of each layer is often handled by separate functions in the IT organization.  DevOps has great agility in cloud applications.  Here is where we see the security exposure on containers and underlying infrastructure.  

Access Administration

Containers are not agile.  Access to the container is access to both container software and code.  Visibility to who has access to what is within the container is not transparent.  Unauthorized installs, abnormal login attempts/failures, key file changes are all examples of insider threats.

Change Management

Orchestration tools have full permissions across all containers.  Changes made to container 1 replicate to all containers on the same node.  Replication of an error or vulnerability may occur rapidly. DevOps can create new services without mapping to container boundaries or segregated access controls.

Kubernetes is designed to orchestrate platform and tool building and works well with container tools such as Docker.  Amazon’s own native container set-up Elastic Compute Cloud (EC2) is designed to orchestrate build, distribution, and running containers.  Security may not be included in managing the container by the DevOps team–such as not updating to use current AMI instance or current versions of orchestration tools such as AWS Internet of Things (IoT) GreenGrass core.

Network Management

Cloud network security has the same dependency as on-premise security for patch and upgrade management.  If the operating systems are not updated with current patches and supported versions, an exploit is feasible.

Vulnerability runC allows malicious containers to overwrite the host runC binary, gain root-level code execution on the host.  If the latest patch has not been applied, like on-premise systems, the exposure may be exploited.

Container Security Platform

The container security settings are designed for efficiency, not necessarily for preventing vulnerability exploits.  The design and management of containers require additional configuration and oversight to reduce exposures.  Across container orchestration tools, we have common security gaps in using container security platforms. 

Last year Docker Hub removed 17 Docker images because they contained cryptocurrency miners.  Do you know where your container images are coming from?

AWS EC2 or Kubernetes-related products such as Aqua, Twistlock, Stackrox, Docker are susceptible to vulnerabilities.  Amazon posted a notice on vulnerabilities found on open-source container management systems.  Their February 13, 2019 alert pointed out vulnerabilities on Linux, ECS, Amazon Kubernetes (EKS), AWS Fargate, and more.  (source: https://aws.amazon.com/security/security-bulletins/AWS-2019-002/).

Mange Container Complexity

Container orchestration relies heavily on your infrastructure.  How does your solution integrate with cloud/on-premise solution?  Are you buying more toolsets to get things done? Consider the following:

  • When applications are mapped to a container IP, the application may change containers causing obsolescence of IP maps.
  • Misconfiguring based on the OS. SELinux containers are difficult to set up correctly as they were designed to run on servers.  DevOps may set it up for functionality, but not security as that is outside their expertise.
  • New services are launched without security management, gaps in namespace, provisioning, mapping and so forth.

Consistency in Patch Management

As the AWS CVE security notice points out, patch management is still required to ensure containers are running the latest versions

Security Mitigation

Review your current orchestration tool configuration settings and be sure to refresh internal policies and procedures for patch management to include container infrastructure components.

  • Know that out of the box vendor software is not inherently secure (K8s API listens on port 8080 with no security checks).
  • Benchmarking security maturity (system access & users, patching & vulnerability mgt, infrastructure control plane, networking, runtime & services).
  • Patching and vulnerability management.
  • Deploy defense in depth-use tools to protect K8s clusters from exposure to the open internet (NGINX for example).  Helps engineers from having to manually provision namespaces each time developers launch a new service.

Containerized applications deployed in the cloud make it easier for organizations to give needed services to their customers more quickly. There continue to be exciting advances in technologies that analyze behavior in real-time to monitor for and alert on misconfigurations and other vulnerabilities in cloud-based container infrastructures.  Coupling these technologies with anomaly analysis and evolved security best practices will create the necessary threat detection, protection, and response controls essential to keeping these dynamic clouds secure.