cd ..

Bare-metal Kubernetes platform

A bare-metal Kubernetes platform with dual-stack networking, Istio, and DNS names and TLS certificates issued automatically.

KuberneteskubesprayMetalLBIstioGateway APIexternal-dnscert-managerstep-ca

Context

Workloads were spread across standalone VMs and Docker hosts, and developers had no shared place to deploy and test. The goal was one Kubernetes platform for both.

What I did

  • Cluster build. Kubernetes on bare metal with kubeadm, using kubespray’s Ansible playbooks as the layer on top.
  • Networking. MetalLB for load balancer IPs, with dual-stack IPv4/IPv6 networking.
  • Ingress and mesh. Istio as both the Gateway API implementation and the service mesh.
  • DNS. external-dns updates Active Directory DNS through the RFC2136 provider, authenticating with Kerberos (GSS-TSIG).
  • Internal PKI. step-ca as the certificate authority, with cert-manager issuing certificates through Smallstep’s step-issuer.
  • Node tuning. Kernel and systemd settings for Kubernetes: sysctl networking (conntrack, forwarding, bridge netfilter), inotify and open-file limits, swap off and cgroup v2 with the systemd driver.

The hard part

Getting DNS and PKI fully automatic. external-dns had to authenticate to AD DNS with Kerberos, and cert-manager had to trust and talk to an internal step-ca. Once both worked, a new service got a DNS name and a valid internal certificate automatically.

Outcome

  • Developers deploy and test on one platform, with DNS names and internal TLS issued automatically.
  • Storage comes from Ceph through Rook (see Rook-Ceph storage).