Why Coinbase Does Not Use Kubernetes to Run Their Container Workloads

Coinbase recently wrote about why Kubernetes is not part of their technology pile. Coinbase uses containers, but they run them in VMs. For deployment, they use Odin, its open source solution for deploying their services in VMs as self-scaled groups. The adoption of Kubernetes adds unnecessary complexity to its current deployment pipeline. In addition, they would prefer to explore other options such as Fargate or ECS before choosing Kubernetes directly. Coinbase stated that Kubernetes is not the right tool for them at the moment.

From a technological standpoint, Kubernetes does not solve any of Coinbase’s customer problems. On the contrary, for them, Kubernetes creates a new set of challenges. For example, they will need to dedicate a team to build the infrastructure necessary to run their services. They will also have to translate their current security practices for Kubernetes. Furthermore, Coinbase said that existing managed services from cloud providers, such as EKS or GKE, are not yet mature enough. For example, if they had to upgrade a cluster, they said they would need a “much more operational-heavy focus” than they currently have.

When someone asked Kelsey Hightower what his thoughts were on Coinbase not using Kubernetes, he replied:

Coinbase builds and maintains their own platform that works for them. Coinbase provided analysis worth studying. The big takeaway for me: asking people to manage their own Kubernetes cluster is like asking people to manage their own hypervisors only when they want VMs.

Coinbase’s existing technology pile consists mostly of containers running in EC2 cases. For application service discovery, they use Route53 in combination with application load balancers and Envoy. They rate their services through self-rating groups (ASGs). They also use lambda functions to organize locations by step functions. Coinbase uses Odin, the orchestral platform they built to use their services such as ASGs, and Codeflow, their in-house tool to manage deployment through UI. Odin has all the logic for doing placements gradually using health checks and can even perform backlogs when needed.

Coinbase defines the specifications of the desired state of their services as an instance type or security groups through a JSON manifesto, somewhat similar to the Kubernetes YAML manifesto. In this respect, Drew Rothstein, engineering director of Coinbase:

We enable the same key features in Kubernetes: deploy button + return in Codeflow, scaling based on defined heuristic ones (we support custom AWS metrics or standard CPU metrics), and rescheduling / moving your containers if your VM will die or become unhealthy in your ASG.

At present, Coinbase has no plans to use Kubernetes to organize their services as they do not need to fulfill a higher usage case for container orchestra. Instead, they would like to evaluate other options before directly adopting Kubernetes. For example, as their infrastructure runs in AWS, they would evaluate managed services like Fargate or ECS. In addition, Odin is already meeting their current needs, and there would be no significant gains for their engineers if they switched to Kubernetes.

Furthermore, Coinbase bases this decision primarily on customer needs, to the point where Rothstein said: “If the barrier to accessing our current platform changed significantly and that is now a clear differentiator , then we would also explore offering a different platform. “Coinbase would prefer a customer-focused reason to implement a change in its orchestra platform. For example, customers’ potential needs might be to reduce deployment times, and have different deployment patterns beyond canary or blue / green, or complex service mesh interactions from what they currently have.

Finally, Coinbase expressed that they do not think Kubernetes is a bad tool; it is not just the tool for them. Coinbase believes Kubernetes is great despite its current challenges, and projects like Knative or Fargate are increasing the level of abstraction to solve many of these challenges.