Kubernetes has become the project to turn to if you need container orchestration at scale. The open source container orchestration system out of Google is well-regarded, well-supported, and evolving fast.
Kubernetes is also sprawling, complex, and difficult to set up and configure. Not only that, but much of the heavy lifting is left to the end user. The best approach, therefore, isn’t to grab the bits and try to go it alone, but to seek out a complete container solution that includes Kubernetes as a supported, maintained component.
Here I’ve listed the 9 most prominent Kubernetes offerings—what amount to distributions that incorporate Kubernetes plus container tools, in the same sense that various vendors offer distributions of the Linux kernel and its userland.
Note that this list does not include dedicated cloud services, such as Amazon EKS or Google Kubernetes Engine, but focuses on software distributions that can be run locally or as a cloud-hosted option.
CoreOS Tectonic/Red Hat CoreOS
CoreOS is the provider of a container-focused Linux distribution, compatible with Docker but with an opinionated image format and runtime of its own, and an “enterprise-grade Kubernetes” distribution. Together they form the foundation of the CoreOS Tectonic stack.
The CoreOS operating system, Container Linux, stands apart chiefly by being delivered as a set of containerized components. This way, automated updates to the OS can be slipstreamed into production without taking down running applications. CoreOS also touts “one-click” updates to Kubernetes. CoreOS Tectonic runs on Amazon Web Services, Microsoft Azure, and bare metal.
Red Hat recently acquired CoreOS, with plans to integrate it into Red Hat OpenShift. Container Linux will be rebranded Red Hat CoreOS. The move isn’t expected to be completed until 2020, but Container Linux will continue to be supported until then. According to Red Hat, “almost all” of CoreOS Tectonic’s features will be available after the transition.
See InfoWorld’s review of CoreOS Container Linux for more details.
Canonical Distribution of Kubernetes
Canonical, makers of Ubuntu Linux, provides its own Kubernetes distribution. One of the big selling points for the Canonical Distribution of Kubernetes is the widely respected, well-understood, and commonly deployed Ubuntu Linux distribution underneath. Canonical claims that its stack will work in any cloud or on-prem deployment, with support included for both CPU- and GPU-powered workloads. Paying customers can have their Kubernetes cluster remotely managed by Canonical engineers.
Canonical’s Kubernetes distribution is also available in a miniature version, Microk8s. Developers and Kubernetes newcomers can install Microk8s on a notebook or desktop, and use it for testing, experimentation, or even production use on low-profile hardware.
Canonical and Rancher Labs (see below) co-produce a product, Cloud Native Platform, that pairs Canonical’s Kubernetes distro with Rancher’s container management platform. The idea is to use Kubernetes to manage the containers running in each cluster, and use Rancher to manage multiple Kubernetes clusters. Cloud Native Platform will become available with Rancher 2.0, which is currently available in a beta preview.
Docker Community Edition / Docker Enterprise
For many of us, Docker is containers. And since 2014, Docker has had its own clustering and orchestration system, Docker Swarm, which until recently stood as a competitor to Kubernetes. Then in October 2017, Docker announced it would be adding Kubernetes—in its unmodified, vanilla state—as a standard pack-in with both Docker Community Edition and Docker Enterprise 2.0 and later editions.
Docker Enterprise 3.0 adds the Docker Kubernetes Service, a Kubernetes integration that keeps versions of Kubernetes consistent between developer desktops and production deployments.
In short, Docker Inc. has read the writing on the container orchestration wall and acknowledged that Kubernetes is more suitable than Swarm for managing large and complex container environments. However, Docker still includes its original clustering system, “swarm mode”, for more modest jobs—for instance, a local, behind-the-firewall application that isn’t likely to grow much, or to maintain existing swarm-mode clusters that don’t need to be revamped.
Heptio Kubernetes Subscription
Two of Kubernetes’s creators, Craig McLuckie and Joe Beda, founded Heptio to offer services and products around Kubernetes. Their first major offering is the Heptio Kubernetes Subscription (HKS), a Kubernetes deployment with paid, 24/7 support provided by Heptio. Pricing starts at $2,000 per month.
The main pitch with Heptio is enterprise-grade Kubernetes without vendor lock-in. Deployments can run on public clouds or private hardware. All the tools offered by Heptio to manage Kubernetes configurations are open source, and fixes are delivered directly to the supported clusters.
VMware acquired Heptio in 2018, but the acquisition has not yet affected plans for Heptio’s product portfolio.
Billed as “Kubernetes that just works,” Kontena Pharos follows roughly the same playbook as Red Hat’s Linux offerings. At bottom it’s a CNCF-certified Kubernetes distribution available under the Apache 2 license (as per Fedora, or CentOS). Those with the money to spend (as per Red Hat Enterprise Linux) can buy professional-level features, consulting, support services, and certain fixed-price offerings such as moving to a cloud-native infrastructure.
The core Pharos distribution comes standard with basic features like automatic security updates and multiple container runtimes. The for-pay product adds enterprise tools, such as the Kontena Lens dashboard, the Kontena Storage distributed storage system, backup, load balancing, and the ability to deploy the cluster in an air-gapped environment.
The professional edition has a thirty-day evaluation period, with support subscriptions that begin at €375 per month. The open source edition has no time limit and no licensing costs.
Pivotal Container Service (PKS)
Pivotal, best known for its work on Cloud Foundry, offers an enterprise-grade Kubernetes called Pivotal Container Service (PKS). PKS draws on many other Pivotal projects for its inspiration. For instance, it uses the Kubo project, which is also used in Pivotal’s Cloud Foundry, to launch and manage Kubernetes clusters.
PKS’s standout feature is close integration with the VMware virtualization stack; in fact PKS is a joint VMware-Pivotal project. Containers running on PKS have access to services normally available only to virtual machines running on vSphere, such as persistent storage in VMware VSAN. In addition, PKS can be managed through VMware Cloud Foundation, used for managing VMware infrastructure in public and private cloud environments.
In short, any shop with an investment in VMware and a growing interest in Kubernetes, may want to look into PKS to make the most of their existing VMware setup.
Rancher Labs has incorporated Kubernetes into its container management platform—called, simply, Rancher—with version 2.0. Rancher 2.0 works on a higher level than other Kubernetes distributions, sitting atop your Linux hosts, Docker containers, and Kubernetes nodes, managing all of them at arm’s length regardless of location or infrastructure. It can even manage Kubernetes clusters on Amazon EKS, Google Kubernetes Engine, Azure Kubernetes Service, and other Kubernetes-as-a-service clouds.
Rancher also comes with its own Kubernetes distribution. Rancher is meant to remove a lot of the drudgery from the process of setting up a Kubernetes cluster and customizing Kubernetes for a specific environment, without allowing those customizations to get in the way of smooth upgrades to Kubernetes—a key consideration for such a fast-moving and constantly updated project.
Rancher also offers a minimal Kubernetes distribution called K3s. Optimized for low-profile deployments, K3s requires a mere 512 MB of RAM per server instance and 200 MB of disk space. It squeezes into this footprint by omitting all legacy, alpha-grade, and nonessential features, as well as many less commonly used plug-ins (although those can be added back in if needed).
Red Hat OpenShift
Red Hat OpenShift, Red Hat’s PaaS product, originally used Heroku buildpack-like “cartridges” to package applications, which were then deployed in containers called “gears.” Then Docker came along, and OpenShift was reworked to make use of the new container image and runtime standard. Inevitably, Red Hat also adopted Kubernetes as the orchestration technology within OpenShift.
OpenShift was built to provide abstraction and automation for all the components in a PaaS. This abstraction and automation also extend to Kubernetes, which still imposes a fair amount of administrative burden, so OpenShift can be used to alleviate that as part of the larger mission of deploying a PaaS.
As noted above, CoreOS Tectonic is being merged into Red Hat OpenShift, although the merger of technologies isn’t expected to be complete until 2020.
See InfoWorld’s review of Red Hat OpenShift 3 for more details.
SUSE Cloud as a Service Platform
Best known for a Linux distribution widely popular in Europe, SUSE also offers the SUSE CaaS Platform. Conceptually, SUSE CaaS Platform is reminiscent of CoreOS Tectonic, combining a bare-metal “micro” OS that runs containers, Kubernetes for container orchestration, a built-in image registry, and cluster configuration tools.
SUSE CaaS Platform 3, released in 2018, added multi-master functionality to make clusters more resilient to a master node crash and a kernel tuning feature for making custom adjustments to the included Linux kernel.
SUSE CaaS Platform can run on public clouds as well as local bare metal, but with the caveat that “SUSE does not currently support any integration into the underlying cloud infrastructure.” Meaning that SUSE CaaS Platform isn’t designed to complement Amazon EKS or Google Kubernetes Engine, but to circumvent them, allowing you to run containers across multiple clouds and data centers.
Gravitational, maker of the Teleport SSH server, also produces Telekube, a “production hardened” Kubernetes distribution that runs on local or remote clusters. Telekube is positioned as a solution for a private SaaS platform, or for running Kubernetes as a service throughout multiple regions or hosting providers.
Apps on Telekube must be prepared to run in containers on Kubernetes. They must also be packaged into “Bundles” that are then published to Kubernetes clusters for distribution. Bundling requires some additional work on top of all of the other preparation needed to deploy container-based applications, but the Bundle manifest is the only Telekube-specific addition that you need to maintain.
Another Gravitational tool for Kubernetes, Gravity, allows you to snapshot an entire Kubernetes cluster—including all of its apps and configuration—and deploy the snapshot to any other Kubernetes environment.