cd ..

Hyper-converged Proxmox and Ceph

Turning a 3-node Proxmox VE cluster into hyper-converged compute and storage: from 30k IOPS on a direct SSD to 80k on Ceph.

Proxmox VECephProxmox SDNiperf3rados benchfio

Context

The Proxmox VE cluster (see the ESXi migration) was converted into a hyper-converged one: compute, storage and networking on the same nodes, with VMs on replicated Ceph storage.

What I did

  • Ceph on the Proxmox nodes. A 3-node Proxmox VE cluster running Ceph. Ceph’s public and cluster traffic share the same physical link, separated on different VLANs.
  • Software-defined networking. Proxmox SDN with simple zones.
  • Validation. Benchmarked every layer: iperf3 for raw network throughput between nodes, rados bench for Ceph itself, and fio for disk performance inside VMs.

Outcome

  • Random IOPS went from about 30k on a direct SSD to 80k on Ceph OSDs.
  • rados bench reached transfer rates of up to 2.8 GB/s.
  • All within a 3-node Proxmox and Ceph cluster.