Kubernetes on Metal

Keidrych Anton-Oates <[email protected]>

Why do This?

  • Performance

  • Cost Savings!

  • Reliance on Hardware Accelerators

    • Field Programmable Gate Array (FGPM)

    • Intel Optimizations

    • Multiple Graphics Cards

  • Ingress on Metal: Dynamic / AB / Canary

Definitions

Kubernetes Illusion

Kubernetes Orchestration

Kubernetes Reality

"Kubernetes…​ eliminates the need for orchestration…​ (it) is composed of a set of independent, composable control processes that continuously drive the current state towards the provided desired state" — What is Kubernetes

Containers

Containers are MicroOS FileSystems supported by Minimal HyperVisors.

KataContainers

MicroServices

“Simply the ability for each of your development teams to work on an independently shippable unit of code”

Or, in technical terms:

“An architecture for the distributed development of applications”

Serverless

'pay-as-you-go’ approach via distributed, short-lived, eventually executed applications

Architecture Overview

Kubernetes

Kubernetes 101

Landscape

CNCF Landscape

CNCF Architecture

Identity Provider

Options

Cloud Provider

Kubernetes must run on a Cloud Provider Fake Cloud Provider

Interaction

Salt, Ansible, & Terraform are all typically used for Cloud Native Provisioning.

Terraform is the only one trustable enough to be controlled by Kubernetes as it just provisions

OVirt

  • Official Cloud Provider

  • Requires Project Atomic

    • Micro Server Management = Way of Life

Digital Rebar

  • Needs 1 DHCP Server per Subnet

  • iPXE boot enabled on machines

  • Terraform support

    • Filtering & Selecting Machines

    • Possible to enable Auto Scaling (on metal)

Image Registry

Options

Volume Plugin

Options

Must be CSI Compliant

Network Plugin

Options

Must be CNI Compliant Must Provide Transparent Encryption

Container Runtime

Options

Layer By Layer

Minimal Decisions

Tenancy

  • Only run Kubernetes for Single Tenants

  • Kubernetes requires a massive overhaul for Multi Tenant support

    • Use separate clusters meshed together if really necessary

Installer

Installer != Distribution

  • Gotcha: CNCF mislabels some distributions as installers.

  • Contain only the essentials to start & upgrade Kubernetes

  • KubeADM is the most advanced.

Components

Components (cont)

Non-Hosted Alternatives

Maximal Decisions

Why no Envoy / Istio?

  • Istio is an independent ecosystem

    • Kubernetes is hard, but standardized

    • Istio is another complex interface & technology

  • BPF & IPv6 > Envoy functionality

  • mTLS is redundant if not multi-tenant.

    • CNI already transparently encrypts Node 2 Node traffic

Non-Hosted Alternatives

Augmented Decisions

Opportunity Costs

Cluster Management

  • Time:

    • Continual

  • People:

    • 3 per cluster

(DIY) Do it Yourself

  • Time:

    • approx 15 months

  • People:

    • 5+ ongoing

Certified Distribution

  • Time:

    • approx 12 months

    • rolling tail of 12 months

  • People:

    • 3 + support

    • rolling tail: 10+

(CaaS) Container as a Service

  • Time:

    • approx 1 month

    • rolling tail: 10% time

  • People: Developers + Operations

No Code — Kelsey Hightower’s view

Hire a Helper

  • Time:

    • within 3 months DevTest

    • within 6 months Production

  • People

    • 1 Helper

    • rolling tail: 1-2 days every quarter

No Substitute for Experience

Operational Platform

  • Time:

    • within 1 week DevTest

    • within 1 month Production

    • rolling tail: 1% time Developers

  • People

    • 1 per cloud (Metal / Public)

Operational Platform Includes

  • Better Scheduler

  • Configuration Management

  • Continuous Delivery

  • Data Center Replacement

  • Dependency Management

  • Federation of Clusters

  • Governance

Finally

“Kubernetes has change-tolerance built into its DNA.” — Cornelia Davis, Senior Director of Technology, Pivotal