Hyper-converged Proxmox and Ceph
Turning a 3-node Proxmox VE cluster into hyper-converged compute and storage: from 30k IOPS on a direct SSD to 80k on Ceph.
Context
The Proxmox VE cluster (see the ESXi migration) was converted into a hyper-converged one: compute, storage and networking on the same nodes, with VMs on replicated Ceph storage.
What I did
- Ceph on the Proxmox nodes. A 3-node Proxmox VE cluster running Ceph. Ceph’s public and cluster traffic share the same physical link, separated on different VLANs.
- Software-defined networking. Proxmox SDN with simple zones.
- Validation. Benchmarked every layer:
iperf3for raw network throughput between nodes,rados benchfor Ceph itself, andfiofor disk performance inside VMs.
Outcome
- Random IOPS went from about 30k on a direct SSD to 80k on Ceph OSDs.
rados benchreached transfer rates of up to 2.8 GB/s.- All within a 3-node Proxmox and Ceph cluster.