Bare-metal Kubernetes platform
A bare-metal Kubernetes platform with dual-stack networking, Istio, and DNS names and TLS certificates issued automatically.
Context
Workloads were spread across standalone VMs and Docker hosts, and developers had no shared place to deploy and test. The goal was one Kubernetes platform for both.
What I did
- Cluster build. Kubernetes on bare metal with kubeadm, using kubespray’s Ansible playbooks as the layer on top.
- Networking. MetalLB for load balancer IPs, with dual-stack IPv4/IPv6 networking.
- Ingress and mesh. Istio as both the Gateway API implementation and the service mesh.
- DNS. external-dns updates Active Directory DNS through the RFC2136 provider, authenticating with Kerberos (GSS-TSIG).
- Internal PKI. step-ca as the certificate authority, with cert-manager issuing certificates through Smallstep’s step-issuer.
- Node tuning. Kernel and systemd settings for Kubernetes: sysctl networking (conntrack, forwarding, bridge netfilter), inotify and open-file limits, swap off and cgroup v2 with the systemd driver.
The hard part
Getting DNS and PKI fully automatic. external-dns had to authenticate to AD DNS with Kerberos, and cert-manager had to trust and talk to an internal step-ca. Once both worked, a new service got a DNS name and a valid internal certificate automatically.
Outcome
- Developers deploy and test on one platform, with DNS names and internal TLS issued automatically.
- Storage comes from Ceph through Rook (see Rook-Ceph storage).