If you have a ceph cluster and you would like to add a rbd pool to your kvm, qemu or libvirt, you can follow the steps listed as below.

Define Variables

We need to define some variables ahead of time so that we can use them during the overall creation and finally provide an automation script to automate the overall process.

Most of the information is referenced from ceph official document but should be refined according to the actual need.

# The number of placement groups used for the storage pool
# It should be refined to fit the storage usage in actual
export CEPH_PGS="128"
export CEPH_USER="libvirt"
export CEPH_POOL="libvirt-pool"
export CEPH_RADOS_HOST="localhost"
export CEPH_RADOS_PORT="6789"
export VIRT_SCRT_UUID="$(uuidgen)"
export VIRT_SCRT_PATH="/tmp/libvirt-secret.xml"
export VIRT_POOL_PATH="/tmp/libvirt-rbd-pool.xml"

Create a Ceph OSD Pool and a Ceph User

First, we need to create a ceph osd pool specially for kvm, qemu, libvirt storage usage.

$ ceph osd pool create ${CEPH_POOL} ${CEPH_PGS} ${CEPH_PGS}

Second, we need a ceph user to manipulate and only restricted to manipulate the ceph osd pool we just created.

$ ceph auth get-or-create "client.${CEPH_USER}" mon "profile rbd" osd "profile rbd pool=${CEPH_POOL}"

Add a Libvirt Secret

We need to add a libvirt secret file for the authentications.

First we need to create the secret file

$ cat > "${VIRT_SCRT_PATH}" <<EOF
<secret ephemeral='no' private='no'>
  <uuid>${VIRT_SCRT_UUID}</uuid>
  <usage type='ceph'>
    <name>client.${CEPH_USER} secret</name>
  </usage>
</secret>
EOF

and then, we embed the ceph user's auth key into the secret file with the uuid specified.

$ virsh secret-define --file "${VIRT_SCRT_PATH}"
$ rm -f "${VIRT_SCRT_PATH}"
$ virsh secret-set-value --secret "${VIRT_SCRT_UUID}" --base64 "$(ceph auth get-key client.${CEPH_USER})"

Define the RBD Storage Pool in Libvirt

We also need to define the RBD storage pool in libvirt as well.

First we need to create the storage pool definition file

$ cat > "${VIRT_POOL_PATH}" <<EOF
<pool type="rbd">
  <name>${CEPH_POOL}</name>
  <source>
    <name>${CEPH_POOL}</name>
    <host name='${CEPH_RADOS_HOST}' port='${CEPH_RADOS_PORT}' />
    <auth username='${CEPH_USER}' type='ceph'>
      <secret uuid='${VIRT_SCRT_UUID}'/>
    </auth>
  </source>
</pool>
EOF

and then, we can apply and start the pool.

$ virsh pool-define "${VIRT_POOL_PATH}"
$ rm -f "${VIRT_POOL_PATH}"
$ virsh pool-autostart "${CEPH_POOL}"
$ virsh pool-start "${CEPH_POOL}"

Create a Test VM via Virt-install Command

To boot with UEFI, we need ovmf installed.

$ apt install ovmf
$ yum install edk2-ovmf

Execute the following script and verify that the Test-VM is booting and installing fine.

#!/bin/bash

export CEPH_POOL="libvirt-pool"

export VM_NAME="Test-VM"
export VM_DEV="vda"
export VM_SZ="8G"
export VM_IMAGE="/var/lib/libvirt/images/ubuntu-18.04.3-live-server-amd64.iso"
export VM_VOLUME="ubuntu-18.04-server"
export VM_CPU=2
export VM_MEM=2048

# Create a volume for the VM
# Legacy qemu-img command:
# qemu-img create -f rbd "rbd:${CEPH_POOL}/${VM_VOLUME}" "${VM_SZ}"
virsh vol-create-as "${CEPH_POOL}" "${VM_VOLUME}" --capacity "${VM_SZ}" --format raw

# Create and install the VM
virt-install                                \
  --connect qemu:///system                  \
  --virt-type kvm                           \
  --name "${VM_NAME}"                       \
  --vcpus="${VM_CPU}" --memory "${VM_MEM}"  \
  --disk "vol=${CEPH_POOL}/${VM_VOLUME}"    \
  --cdrom "${VM_IMAGE}"                     \
  --boot uefi                               \
  --noautoconsole                           \
  --os-variant "ubuntu18.04"                \
  --autostart

Automation Script

To sum up, we can automate the overall process as following:

#!/bin/bash

# file: libvirt-add-rbd-storage.bash

# The number of placement groups used for the storage pool
# It should be refined to fit the storage usage in actual
export CEPH_PGS="128"
export CEPH_USER="libvirt"
export CEPH_POOL="libvirt-pool"
export CEPH_RADOS_HOST="localhost"
export CEPH_RADOS_PORT="6789"
export VIRT_SCRT_UUID="$(uuidgen)"
export VIRT_SCRT_PATH="/tmp/libvirt-secret.xml"
export VIRT_POOL_PATH="/tmp/libvirt-rbd-pool.xml"

# Create a Ceph OSD Pool
ceph osd pool create ${CEPH_POOL} ${CEPH_PGS} ${CEPH_PGS}
# Create a Ceph User
ceph auth get-or-create "client.${CEPH_USER}" mon "profile rbd" osd "profile rbd pool=${CEPH_POOL}"

# Create a Libvirt Secret File
cat > "${VIRT_SCRT_PATH}" <<EOF
<secret ephemeral='no' private='no'>
  <uuid>${VIRT_SCRT_UUID}</uuid>
  <usage type='ceph'>
    <name>client.${CEPH_USER} secret</name>
  </usage>
</secret>
EOF
virsh secret-define --file "${VIRT_SCRT_PATH}"
rm -f "${VIRT_SCRT_PATH}"
# Embed the ceph user's auth key
virsh secret-set-value --secret "${VIRT_SCRT_UUID}" --base64 "$(ceph auth get-key client.${CEPH_USER})"

# Define the RBD Storage Pool in Libvirt
cat > "${VIRT_POOL_PATH}" <<EOF
<pool type="rbd">
  <name>${CEPH_POOL}</name>
  <source>
    <name>${CEPH_POOL}</name>
    <host name='${CEPH_RADOS_HOST}' port='${CEPH_RADOS_PORT}' />
    <auth username='${CEPH_USER}' type='ceph'>
      <secret uuid='${VIRT_SCRT_UUID}'/>
    </auth>
  </source>
</pool>
EOF
virsh pool-define "${VIRT_POOL_PATH}"
rm -f "${VIRT_POOL_PATH}"

# Start the storage pool
virsh pool-autostart "${CEPH_POOL}"
virsh pool-start "${CEPH_POOL}"