Kubernetes on Metal

Keidrych Anton-Oates <[email protected]>

Why do This?

  • Performance

  • Cost Savings!

  • Reliance on Hardware Accelerators

    • Field Programmable Gate Array (FGPM)

    • Intel Optimizations

    • Multiple Graphics Cards

  • Ingress on Metal: Dynamic / AB / Canary


Kubernetes Illusion

Kubernetes Orchestration

Kubernetes Reality

"Kubernetes…​ eliminates the need for orchestration…​ (it) is composed of a set of independent, composable control processes that continuously drive the current state towards the provided desired state" — What is Kubernetes


Containers are MicroOS FileSystems supported by Minimal HyperVisors.



“Simply the ability for each of your development teams to work on an independently shippable unit of code”

Or, in technical terms:

“An architecture for the distributed development of applications”


'pay-as-you-go’ approach via distributed, short-lived, eventually executed applications

Architecture Overview


Kubernetes 101


CNCF Landscape

CNCF Architecture

Identity Provider


Cloud Provider

Kubernetes must run on a Cloud Provider Fake Cloud Provider


Salt, Ansible, & Terraform are all typically used for Cloud Native Provisioning.

Terraform is the only one trustable enough to be controlled by Kubernetes as it just provisions


  • Official Cloud Provider

  • Requires Project Atomic

    • Micro Server Management = Way of Life

Digital Rebar

  • Needs 1 DHCP Server per Subnet

  • iPXE boot enabled on machines

  • Terraform support

    • Filtering & Selecting Machines

    • Possible to enable Auto Scaling (on metal)

Image Registry


Volume Plugin


Must be CSI Compliant

Network Plugin


Must be CNI Compliant Must Provide Transparent Encryption

Container Runtime


Layer By Layer

Minimal Decisions


  • Only run Kubernetes for Single Tenants

  • Kubernetes requires a massive overhaul for Multi Tenant support

    • Use separate clusters meshed together if really necessary


Installer != Distribution

  • Gotcha: CNCF mislabels some distributions as installers.

  • Contain only the essentials to start & upgrade Kubernetes

  • KubeADM is the most advanced.


Components (cont)

Non-Hosted Alternatives

Maximal Decisions

Why no Envoy / Istio?

  • Istio is an independent ecosystem

    • Kubernetes is hard, but standardized

    • Istio is another complex interface & technology

  • BPF & IPv6 > Envoy functionality

  • mTLS is redundant if not multi-tenant.

    • CNI already transparently encrypts Node 2 Node traffic

Non-Hosted Alternatives

Augmented Decisions

Opportunity Costs

Cluster Management

  • Time:

    • Continual

  • People:

    • 3 per cluster

(DIY) Do it Yourself

  • Time:

    • approx 15 months

  • People:

    • 5+ ongoing

Certified Distribution

  • Time:

    • approx 12 months

    • rolling tail of 12 months

  • People:

    • 3 + support

    • rolling tail: 10+

(CaaS) Container as a Service

  • Time:

    • approx 1 month

    • rolling tail: 10% time

  • People: Developers + Operations

No Code — Kelsey Hightower’s view

Hire a Helper

  • Time:

    • within 3 months DevTest

    • within 6 months Production

  • People

    • 1 Helper

    • rolling tail: 1-2 days every quarter

No Substitute for Experience

Operational Platform

  • Time:

    • within 1 week DevTest

    • within 1 month Production

    • rolling tail: 1% time Developers

  • People

    • 1 per cloud (Metal / Public)

Operational Platform Includes

  • Better Scheduler

  • Configuration Management

  • Continuous Delivery

  • Data Center Replacement

  • Dependency Management

  • Federation of Clusters

  • Governance


“Kubernetes has change-tolerance built into its DNA.” — Cornelia Davis, Senior Director of Technology, Pivotal