Ceph Nautilus Lab

Summary⌗

This is the starting point for future ceph labs or test. It is designed with a mixture of drive sizes to allow for different labs and scenarios.

It should take around an hour to build from scratch using the quick setup scripts

Setup VMs⌗

There will be 13 VMs set up and 2 networks. The Detailed Setup shows the full setup on 1 OSD node. Using the Quick Setup script will create the environment from scratch.

Base System Requirements (Does not include optional nodes)

CPU >= 44
Memory >= 82GB
Disk >= 580GB

Host	Role	Count	vCPU	Memory	Disk Size	OSD Disks	OSD Disk Size	Optional
bastion	bastion	1	2	2GB	40GB	0		No
grafana	grafana	1	2	4GB	40GB	0		No
monitor	mon/mgr	3	4	4GB	40GB	0		No
t1-osd	osd	4	4	8GB	40GB	4	5GB	No
t2-osd	osd	4	4	8GB	40GB	4	10GB	No
rgw	rgw	2	2	4GB	40GB	0		Yes
mds	mds	2	4	8GB	40GB	0		Yes
iscsi	iscsi	2	4	8GB	40GB	0		Yes

Detailed Setup⌗

Create Networks⌗

Ceph presentation network

xml ceph-presentation.xml


<network>
  <name>ceph-presentation</name>
  <bridge name="virbr3300"/>
  <forward mode="nat"/>
  <domain name="ceph.lab"/>
  <ip address="10.44.20.1" netmask="255.255.255.0">
    <dhcp>
      <range start="10.44.20.200" end="10.44.20.210"/>
    </dhcp>
  </ip>
</network>

Ceph replication network

xml ceph-replication.xml


<network>
  <name>ceph-replication</name>
  <bridge name="virbr3301"/>
  <ip address="172.16.20.1" netmask="255.255.255.0">
    <dhcp>
      <range start="172.16.20.200" end="172.16.20.210"/>
    </dhcp>
  </ip>
</network>

Create these in libvirt

bash



virsh net-define ceph-presentation.xml 

virsh net-start ceph-presentation

virsh net-autostart ceph-presentation 

virsh net-define ceph-replication.xml 

virsh net-start ceph-replication 

virsh net-autostart ceph-replication

Create VM Example⌗

This will create an OSD node. For other nodes, there wont be the need for as many drives to be created.

Create the OS drive for the node

bash


qemu-img create -f qcow2 /var/lib/libvirt/images/ceph-osd-t1-node01.qcow2 40G

Expand the OS base image into the drive. For this setup it will be using CentOS 7

bash


virt-resize --expand /dev/sda1 /var/lib/libvirt/images/iso/CentOS-7-x86_64-GenericCloud.qcow2 /var/lib/libvirt/images/ceph-osd-t1-node01.qcow2

Customise the OS so it can be used

bash


virt-customize -a /var/lib/libvirt/images/ceph-osd-t1-node01.qcow2 \
  --root-password password:password \
  --uninstall cloud-init \
  --hostname ceph-osd-t1-node01 \
  --ssh-inject root:file:/root/.ssh/id_ed25519.pub \
  --selinux-relabel

Create the 4 OSD drives (5GB for t1 nodes)

bash


for i in `seq -w 01 04`; do 
  qemu-img create -f qcow2 /var/lib/libvirt/images/ceph-osd-t1-node01-osd$i.qcow2 40G;
done

Define the VM, with both networks and all drives attached. Remove --dry-run and --print-xml in order to create the domain.

bash


virt-install --name ceph-osd-t1-node01.ceph.lab \
  --virt-type kvm \
  --memory 16384 \
  --vcpus 4 \
  --boot hd,menu=on \
  --disk path=/var/lib/libvirt/images/ceph-osd-t1-node01.qcow2,device=disk \
  --disk path=/var/lib/libvirt/images/ceph-osd-t1-node01-osd01.qcow2,device=disk \
  --disk path=/var/lib/libvirt/images/ceph-osd-t1-node01-osd02.qcow2,device=disk \
  --disk path=/var/lib/libvirt/images/ceph-osd-t1-node01-osd03.qcow2,device=disk \
  --disk path=/var/lib/libvirt/images/ceph-osd-t1-node01-osd04.qcow2,device=disk \
  --os-type Linux \
  --os-variant centos7 \
  --network network:ceph-presentation \
  --network network:ceph-replication \
  --graphics spice \
  --noautoconsole \
  --dry-run \
  --print-xml

Quick Setup⌗

Script options are set as variables. By default it won’t build any of the optional nodes. If the vars are set to yes this will be build them. It is somewhat idempotent as well. A teardown script is also available to clean this all up.

The nodes built by the script (including optional)

Hostname	Public IP	Replication IP
bastion.ceph.lab	DHCP	None
grafana.ceph.lab	DHCP	None
ceph-mon01.ceph.lab	10.44.20.21	172.16.20.21
ceph-mon02.ceph.lab	10.44.20.22	172.16.20.22
ceph-mon03.ceph.lab	10.44.20.23	172.16.20.23
ceph-t1-osd01.ceph.lab	10.44.20.31	172.16.20.31
ceph-t1-osd02.ceph.lab	10.44.20.32	172.16.20.32
ceph-t1-osd03.ceph.lab	10.44.20.33	172.16.20.33
ceph-t1-osd04.ceph.lab	10.44.20.34	172.16.20.34
ceph-t2-osd01.ceph.lab	10.44.20.41	172.16.20.41
ceph-t2-osd02.ceph.lab	10.44.20.42	172.16.20.42
ceph-t2-osd03.ceph.lab	10.44.20.43	172.16.20.43
ceph-t2-osd04.ceph.lab	10.44.20.44	172.16.20.44
ceph-rgw01.ceph.lab	10.44.20.111	None
ceph-rgw02.ceph.lab	10.44.20.112	None
ceph-mds01.ceph.lab	10.44.20.121	None
ceph-mds02.ceph.lab	10.44.20.122	None
ceph-iscsi01.ceph.lab	10.44.20.131	None
ceph-iscsi02.ceph.lab	10.44.20.132	None

Scripts can also be found on GitHub

bash setup.sh


#!/bin/bash

# Node building vars
image_dir="/var/lib/libvirt/images"
base_os_img="/var/lib/libvirt/images/iso/CentOS-7-x86_64-GenericCloud.qcow2"
ssh_pub_key="/root/.ssh/id_ed25519.pub"

# Network Vars
dns_domain="ceph.lab"

# Extra Vars
root_password="password"
os_drive_size="40G"
tmp_dir="/tmp"

# Ceph Extra nodes 
rgw=no
mds=no
iscsi=no


##### Start #####

# Exit on any failure

set -e

# Create Network files

echo "Creating ceph-presentation xml file"

cat <<EOF > $tmp_dir/ceph-presentation.xml
<network>
  <name>ceph-presentation</name>
  <bridge name="virbr3300"/>
  <forward mode="nat"/>
  <domain name="ceph.lab"/>
  <ip address="10.44.20.1" netmask="255.255.255.0">
    <dhcp>
      <range start="10.44.20.200" end="10.44.20.210"/>
    </dhcp>
  </ip>
</network>
EOF

echo "Creating ceph-replicaton xml file"
cat <<EOF >$tmp_dir/ceph-replication.xml
<network>
  <name>ceph-replication</name>
  <bridge name="virbr3301"/>
  <ip address="172.16.20.1" netmask="255.255.255.0">
    <dhcp>
      <range start="172.16.20.200" end="172.16.20.210"/>
    </dhcp>
  </ip>
</network>
EOF

echo "Creating Ceph networks in libvirt"

check_rep=$(virsh net-list --all | grep ceph-replication >/dev/null && echo "0" || echo "1")
check_pres=$(virsh net-list --all | grep ceph-presentation >/dev/null && echo "0" || echo "1")

networks=()

if [[ $check_rep == "1" ]]; then
  networks+=("ceph-replication")
fi

if [[ $check_pres == "1" ]]; then
  networks+=("ceph-presentation")
fi

net_len=$(echo "${#networks[@]}")

echo "Creating networks ${#networks[@]}"

if [ "$net_len" -ge 1 ]; then
  for network in ${networks[@]}; do 
    virsh net-define $tmp_dir/$network.xml
    virsh net-start $network
    virsh net-autostart $network
  done
fi

# Check OS image exists

if [ -f "$base_os_img" ]; then
  echo "Base OS image exists"
else
  echo "Base image doesn't exist ($base_os_img). Exiting"
  exit 1
fi

# Build OS drives for machines

echo "Starting build of VMs"

echo "Building Bastion & Grafana drives"

for node in bastion grafana; do 
  check=$(virsh list --all | grep $node.$dns_domain > /dev/null && echo "0" || echo "1" )
  if [[ $check == "0" ]]; then
    echo "$node.$dns_domain exists"
  else
    echo "Starting $node"
    echo "Creating $image_dir/$node.$dns_domain.qcow2 at $os_drive_size"
    qemu-img create -f qcow2 $image_dir/$node.$dns_domain.qcow2 $os_drive_size
    echo "Resizing base OS image"
    virt-resize --expand /dev/sda1 $base_os_img $image_dir/$node.$dns_domain.qcow2
    echo "Customising OS for $node"
    virt-customize -a $image_dir/$node.$dns_domain.qcow2 \
      --root-password password:$root_password \
      --uninstall cloud-init \
      --hostname $node.$dns_domain \
      --ssh-inject root:file:$ssh_pub_key \
      --selinux-relabel
  fi
done

check=$(virsh list --all | grep bastion.$dns_domain > /dev/null && echo "0" || echo "1" )
if [[ $check == "1" ]]; then
  echo "Defining Bastion VM"
  virt-install --name bastion.$dns_domain \
    --virt-type kvm \
    --memory 2048 \
    --vcpus 2 \
    --boot hd,menu=on \
    --disk path=$image_dir/bastion.$dns_domain.qcow2,device=disk \
    --os-type Linux \
    --os-variant centos7 \
    --network network:ceph-presentation \
    --graphics spice \
    --noautoconsole
fi

check=$(virsh list --all | grep grafana.$dns_domain > /dev/null && echo "0" || echo "1" )
if [[ $check == "1" ]]; then
  echo "Defining Grafana VM"
  virt-install --name grafana.$dns_domain \
    --virt-type kvm \
    --memory 4096 \
    --vcpus 2 \
    --boot hd,menu=on \
    --disk path=$image_dir/grafana.$dns_domain.qcow2,device=disk \
    --os-type Linux \
    --os-variant centos7 \
    --network network:ceph-presentation \
    --graphics spice \
    --noautoconsole
fi

echo "Building Monitor VMs"

count=1

for mon in `seq -w 01 03`; do 
  check=$(virsh list --all | grep ceph-mon$mon.$dns_domain > /dev/null && echo "0" || echo "1" )
  if [[ $check == "0" ]]; then
    echo "ceph-mon$mon.$dns_domain already exists"
    count=$(( $count + 1 ))
  else
    echo "Creating eth0 ifcfg file"
    mkdir -p $tmp_dir/ceph-mon$mon
    cat <<EOF > $tmp_dir/ceph-mon$mon/ifcfg-eth0
TYPE=Ethernet
NAME=eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.44.20.2$count
NETMASK=255.255.255.0
GATEWAY=10.44.20.1
DNS1=10.44.20.1
ONBOOT=yes
DEFROUTE=yes
EOF
    echo "Creating eth1 ifcfg file"
    cat <<EOF > $tmp_dir/ceph-mon$mon/ifcfg-eth1
TYPE=Ethernet
NAME=eth1
DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.20.2$count
NETMASK=255.255.255.0
EOF
    echo "Starting ceph-mon$mon"
    echo "Creating $image_dir/ceph-mon$mon.$dns_domain.qcow2 at $os_drive_size"
    qemu-img create -f qcow2 $image_dir/ceph-mon$mon.$dns_domain.qcow2 $os_drive_size
    echo "Resizing base OS image"
    virt-resize --expand /dev/sda1 $base_os_img $image_dir/ceph-mon$mon.$dns_domain.qcow2
    echo "Customising OS for ceph-mon$mon"
    virt-customize -a $image_dir/ceph-mon$mon.$dns_domain.qcow2 \
      --root-password password:$root_password \
      --uninstall cloud-init \
      --hostname ceph-mon$mon \
      --ssh-inject root:file:$ssh_pub_key \
      --copy-in $tmp_dir/ceph-mon$mon/ifcfg-eth0:/etc/sysconfig/network-scripts/ \
      --copy-in $tmp_dir/ceph-mon$mon/ifcfg-eth1:/etc/sysconfig/network-scripts/ \
      --selinux-relabel
    echo "Defining ceph-mon$mon.$dns_domain"
    virt-install --name ceph-mon$mon.$dns_domain \
      --virt-type kvm \
      --memory 4096 \
      --vcpus 4 \
      --boot hd,menu=on \
      --disk path=$image_dir/ceph-mon$mon.$dns_domain.qcow2,device=disk \
      --os-type Linux \
      --os-variant centos7 \
      --network network:ceph-presentation \
      --network network:ceph-replication \
      --graphics spice \
      --noautoconsole
    count=$(( $count + 1 ))
  fi
done

echo "Building OSD T1 drives"

count=1

for i in `seq -w 01 04`; do 
  check=$(virsh list --all | grep ceph-t1-osd$i.$dns_domain > /dev/null && echo "0" || echo "1" )
  if [[ $check == "0" ]]; then
    echo "ceph-t1-osd$i.$dns_domain already exists"
    count=$(( $count + 1 ))
  else
    echo "Creating eth0 ifcfg file"
    mkdir -p $tmp_dir/ceph-t1-osd$i
    cat <<EOF > $tmp_dir/ceph-t1-osd$i/ifcfg-eth0
TYPE=Ethernet
NAME=eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.44.20.3$count
NETMASK=255.255.255.0
GATEWAY=10.44.20.1
DNS1=10.44.20.1
ONBOOT=yes
DEFROUTE=yes
EOF
    echo "Creating eth1 ifcfg file"
    cat <<EOF > $tmp_dir/ceph-t1-osd$i/ifcfg-eth1
TYPE=Ethernet
NAME=eth1
DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.20.3$count
NETMASK=255.255.255.0
EOF
    echo "Starting ceph-t1-osd$i"
    echo "Creating $image_dir/ceph-t1-osd$i.$dns_domain.qcow2 at $os_drive_size"
    qemu-img create -f qcow2 $image_dir/ceph-t1-osd$i.$dns_domain.qcow2 $os_drive_size
    for c in {1..4}; do 
      qemu-img create -f qcow2 $image_dir/ceph-t1-osd$i-disk$c.$dns_domain.qcow2 5G
    done
    echo "Resizing base OS image"
    virt-resize --expand /dev/sda1 $base_os_img $image_dir/ceph-t1-osd$i.$dns_domain.qcow2
    echo "Customising OS for ceph-t1-osd$i"
    virt-customize -a $image_dir/ceph-t1-osd$i.$dns_domain.qcow2 \
      --root-password password:$root_password \
      --uninstall cloud-init \
      --hostname ceph-t1-osd$i \
      --ssh-inject root:file:$ssh_pub_key \
      --copy-in $tmp_dir/ceph-t1-osd$i/ifcfg-eth0:/etc/sysconfig/network-scripts/ \
      --copy-in $tmp_dir/ceph-t1-osd$i/ifcfg-eth1:/etc/sysconfig/network-scripts/ \
      --selinux-relabel
    echo "Defining ceph-t1-osd$i"
    virt-install --name ceph-t1-osd$i.$dns_domain \
      --virt-type kvm \
      --memory 8192 \
      --vcpus 4 \
      --boot hd,menu=on \
      --disk path=$image_dir/ceph-t1-osd$i.$dns_domain.qcow2,device=disk \
      --disk path=$image_dir/ceph-t1-osd$i-disk1.$dns_domain.qcow2,device=disk \
      --disk path=$image_dir/ceph-t1-osd$i-disk2.$dns_domain.qcow2,device=disk \
      --disk path=$image_dir/ceph-t1-osd$i-disk3.$dns_domain.qcow2,device=disk \
      --disk path=$image_dir/ceph-t1-osd$i-disk4.$dns_domain.qcow2,device=disk \
      --os-type Linux \
      --os-variant centos7 \
      --network network:ceph-presentation \
      --network network:ceph-replication \
      --graphics spice \
      --noautoconsole
    
    count=$(( $count + 1 ))
  fi
done

echo "Building OSD T2 drives"

count=1

for i in `seq -w 01 04`; do 
  check=$(virsh list --all | grep ceph-t2-osd$i.$dns_domain > /dev/null && echo "0" || echo "1" )
  if [[ $check == "0" ]]; then
    echo "ceph-t2-osd$i.$dns_domain already exists"
    count=$(( $count + 1 ))
  else
    echo "Creating eth0 ifcfg file"
    mkdir -p $tmp_dir/ceph-t2-osd$i
    cat <<EOF > $tmp_dir/ceph-t2-osd$i/ifcfg-eth0
TYPE=Ethernet
NAME=eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.44.20.4$count
NETMASK=255.255.255.0
GATEWAY=10.44.20.1
DNS1=10.44.20.1
ONBOOT=yes
DEFROUTE=yes
EOF
    echo "Creating eth1 ifcfg file"
    cat <<EOF > $tmp_dir/ceph-t2-osd$i/ifcfg-eth1
TYPE=Ethernet
NAME=eth1
DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.20.4$count
NETMASK=255.255.255.0
EOF
    echo "Starting ceph-t2-osd$i"
    echo "Creating $image_dir/ceph-t2-osd$i.$dns_domain.qcow2 at $os_drive_size"
    qemu-img create -f qcow2 $image_dir/ceph-t2-osd$i.$dns_domain.qcow2 $os_drive_size
    for c in {1..4}; do
      qemu-img create -f qcow2 $image_dir/ceph-t2-osd$i-disk$c.$dns_domain.qcow2 10G
    done
    echo "Resizing base OS image"
    virt-resize --expand /dev/sda1 $base_os_img $image_dir/ceph-t2-osd$i.$dns_domain.qcow2
    echo "Customising OS for ceph-t2-osd$i"
    virt-customize -a $image_dir/ceph-t2-osd$i.$dns_domain.qcow2 \
      --root-password password:$root_password \
      --uninstall cloud-init \
      --hostname ceph-t2-osd$i \
      --ssh-inject root:file:$ssh_pub_key \
      --copy-in $tmp_dir/ceph-t2-osd$i/ifcfg-eth0:/etc/sysconfig/network-scripts/ \
      --copy-in $tmp_dir/ceph-t2-osd$i/ifcfg-eth1:/etc/sysconfig/network-scripts/ \
      --selinux-relabel
  
    echo "Defining ceph-t2-osd$i"
    virt-install --name ceph-t2-osd$i.$dns_domain \
      --virt-type kvm \
      --memory 8192 \
      --vcpus 4 \
      --boot hd,menu=on \
      --disk path=$image_dir/ceph-t2-osd$i.$dns_domain.qcow2,device=disk \
      --disk path=$image_dir/ceph-t2-osd$i-disk1.$dns_domain.qcow2,device=disk \
      --disk path=$image_dir/ceph-t2-osd$i-disk2.$dns_domain.qcow2,device=disk \
      --disk path=$image_dir/ceph-t2-osd$i-disk3.$dns_domain.qcow2,device=disk \
      --disk path=$image_dir/ceph-t2-osd$i-disk4.$dns_domain.qcow2,device=disk \
      --os-type Linux \
      --os-variant centos7 \
      --network network:ceph-presentation \
      --network network:ceph-replication \
      --graphics spice \
      --noautoconsole
    count=$(( $count + 1 ))
  fi
done

## Build extra ceph nodes if defines

# If rgw is "yes"

if [[ $rgw == "yes" ]]; then
  count=1
  for rgw in `seq -w 01 02`; do 
    check=$(virsh list --all | grep ceph-rgw$rgw.$dns_domain > /dev/null && echo "0" || echo "1" )
    if [[ $check == "0" ]]; then
      echo "ceph-rgw$rgw.$dns_domain already exists"
      count=$(( $count + 1 ))
    else
      echo "Creating eth0 ifcfg file"
      mkdir -p $tmp_dir/ceph-rgw$rgw
      cat <<EOF > $tmp_dir/ceph-rgw$rgw/ifcfg-eth0
TYPE=Ethernet
NAME=eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.44.20.11$count
NETMASK=255.255.255.0
GATEWAY=10.44.20.1
DNS1=10.44.20.1
ONBOOT=yes
DEFROUTE=yes
EOF
      echo "Starting ceph-rgw$rgw"
      echo "Creating $image_dir/ceph-rgw$rgw.$dns_domain.qcow2 at $os_drive_size"
      qemu-img create -f qcow2 $image_dir/ceph-rgw$rgw.$dns_domain.qcow2 $os_drive_size
      echo "Resizing base OS image"
      virt-resize --expand /dev/sda1 $base_os_img $image_dir/ceph-rgw$rgw.$dns_domain.qcow2
      echo "Customising OS for ceph-rgw$rgw"
      virt-customize -a $image_dir/ceph-rgw$rgw.$dns_domain.qcow2 \
        --root-password password:$root_password \
        --uninstall cloud-init \
        --hostname ceph-rgw$rgw \
        --ssh-inject root:file:$ssh_pub_key \
        --copy-in $tmp_dir/ceph-rgw$rgw/ifcfg-eth0:/etc/sysconfig/network-scripts/ \
        --selinux-relabel
      echo "Defining ceph-rgw$rgw.$dns_domain"
      virt-install --name ceph-rgw$rgw.$dns_domain \
        --virt-type kvm \
        --memory 4096 \
        --vcpus 2 \
        --boot hd,menu=on \
        --disk path=$image_dir/ceph-rgw$rgw.$dns_domain.qcow2,device=disk \
        --os-type Linux \
        --os-variant centos7 \
        --network network:ceph-presentation \
        --graphics spice \
        --noautoconsole
      count=$(( $count + 1 ))
    fi
  done
fi

# If mds set to "yes"

if [[ $mds == "yes" ]]; then
  count=1
  for mds in `seq -w 01 02`; do
    check=$(virsh list --all | grep ceph-mds$mds.$dns_domain > /dev/null && echo "0" || echo "1" )
    if [[ $check == "0" ]]; then
      echo "ceph-mds$mds.$dns_domain already exists"
      count=$(( $count + 1 ))
    else
      echo "Creating eth0 ifcfg file"
      mkdir -p $tmp_dir/ceph-mds$mds
      cat <<EOF > $tmp_dir/ceph-mds$mds/ifcfg-eth0
TYPE=Ethernet
NAME=eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.44.20.12$count
NETMASK=255.255.255.0
GATEWAY=10.44.20.1
DNS1=10.44.20.1
ONBOOT=yes
DEFROUTE=yes
EOF
      echo "Starting ceph-mds$mds"
      echo "Creating $image_dir/ceph-mds$mds.$dns_domain.qcow2 at $os_drive_size"
      qemu-img create -f qcow2 $image_dir/ceph-mds$mds.$dns_domain.qcow2 $os_drive_size
      echo "Resizing base OS image"
      virt-resize --expand /dev/sda1 $base_os_img $image_dir/ceph-mds$mds.$dns_domain.qcow2
      echo "Customising OS for ceph-mds$mds"
      virt-customize -a $image_dir/ceph-mds$mds.$dns_domain.qcow2 \
        --root-password password:$root_password \
        --uninstall cloud-init \
        --hostname ceph-mds$mds \
        --ssh-inject root:file:$ssh_pub_key \
        --copy-in $tmp_dir/ceph-mds$mds/ifcfg-eth0:/etc/sysconfig/network-scripts/ \
        --selinux-relabel
      echo "Defining ceph-mds$mds.$dns_domain"
      virt-install --name ceph-mds$mds.$dns_domain \
        --virt-type kvm \
        --memory 8192 \
        --vcpus 4 \
        --boot hd,menu=on \
        --disk path=$image_dir/ceph-mds$mds.$dns_domain.qcow2,device=disk \
        --os-type Linux \
        --os-variant centos7 \
        --network network:ceph-presentation \
        --graphics spice \
        --noautoconsole
      count=$(( $count + 1 ))
    fi
  done
fi

# If iscsi set to "yes"

if [[ $iscsi == "yes" ]]; then
  count=1
  for iscsi in `seq -w 01 02`; do
    check=$(virsh list --all | grep ceph-iscsi$iscsi.$dns_domain > /dev/null && echo "0" || echo "1" )
    if [[ $check == "0" ]]; then
      echo "ceph-iscsi$iscsi.$dns_domain already exists"
      count=$(( $count + 1 ))
    else
      echo "Creating eth0 ifcfg file"
      mkdir -p $tmp_dir/ceph-iscsi$iscsi
      cat <<EOF > $tmp_dir/ceph-iscsi$iscsi/ifcfg-eth0
TYPE=Ethernet
NAME=eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.44.20.13$count
NETMASK=255.255.255.0
GATEWAY=10.44.20.1
DNS1=10.44.20.1
ONBOOT=yes
DEFROUTE=yes
EOF
      echo "Starting ceph-iscsi$iscsi"
      echo "Creating $image_dir/ceph-iscsi$iscsi.$dns_domain.qcow2 at $os_drive_size"
      qemu-img create -f qcow2 $image_dir/ceph-iscsi$iscsi.$dns_domain.qcow2 $os_drive_size
      echo "Resizing base OS image"
      virt-resize --expand /dev/sda1 $base_os_img $image_dir/ceph-iscsi$iscsi.$dns_domain.qcow2
      echo "Customising OS for ceph-iscsi$iscsi"
      virt-customize -a $image_dir/ceph-iscsi$iscsi.$dns_domain.qcow2 \
        --root-password password:$root_password \
        --uninstall cloud-init \
        --hostname ceph-iscsi$iscsi \
        --ssh-inject root:file:$ssh_pub_key \
        --copy-in $tmp_dir/ceph-iscsi$iscsi/ifcfg-eth0:/etc/sysconfig/network-scripts/ \
        --selinux-relabel
      echo "Defining ceph-iscsi$iscsi.$dns_domain"
      virt-install --name ceph-iscsi$iscsi.$dns_domain \
        --virt-type kvm \
        --memory 8192 \
        --vcpus 4 \
        --boot hd,menu=on \
        --disk path=$image_dir/ceph-iscsi$iscsi.$dns_domain.qcow2,device=disk \
        --os-type Linux \
        --os-variant centos7 \
        --network network:ceph-presentation \
        --graphics spice \
        --noautoconsole
      count=$(( $count + 1 ))
    fi
  done
fi

# Print running VMs

virsh list

Demo⌗

Adding More Disks⌗

If there’s a capacity or a need to add some more drives to the OSD nodes this example will add more drives to the OSD VMs

{f..g} will add 2 more drives to dev/vdf and /def/vdg. Change this to add more.

bash



path=/var/lib/libvirt/images/
size=50G
server=$(virsh list | grep osd | awk '{print $2}')

for s in $server; do \
   d=1; \
  for l in {f..g}; do \
  qemu-img create -f qcow2 $path/$s-extradisk$d.qcow2 $size; \
  virsh attach-disk $s $path/$s-extradisk$d.qcow2 vd$l --subdriver qcow2 --persistent; \
  d=$(( $d + 1 )); \
  done; \
done

Cleanup⌗

Cleanup bash script to remove all the parts of the Ceph lab

Scripts can also be found on GitHub

bash teardown.sh



#!/bin/bash

# Node building vars
image_dir="/var/lib/libvirt/images"
base_os_img="/var/lib/libvirt/images/iso/CentOS-7-x86_64-GenericCloud.qcow2"
ssh_pub_key="/root/.ssh/id_ed25519.pub"

# Network Vars
dns_domain="ceph.lab"

# Extra Vars
root_password="password"
os_drive_size="40G"
tmp_dir="/tmp"


##### Start #####

# Destroy & Undefine all nodes

echo "Destroy & Undefine all nodes"
virsh destroy bastion.$dns_domain
virsh destroy grafana.$dns_domain
virsh undefine bastion.$dns_domain --remove-all-storage
virsh undefine grafana.$dns_domain --remove-all-storage

echo "Removing Monitor nodes"
for mon in `seq -w 01 03`; do
  virsh destroy ceph-mon$mon.$dns_domain
  virsh undefine ceph-mon$mon.$dns_domain --remove-all-storage
done

echo "Removing OSD nodes"
for i in `seq -w 01 04`; do
  virsh destroy ceph-t1-osd$i.$dns_domain
  virsh destroy ceph-t2-osd$i.$dns_domain
  virsh undefine ceph-t1-osd$i.$dns_domain --remove-all-storage
  virsh undefine ceph-t2-osd$i.$dns_domain --remove-all-storage
done

echo "Removing other nodes"
for i in `seq -w 01 02`; do
  virsh destroy ceph-rgw$i.$dns_domain
  virsh destroy ceph-mds$i.$dns_domain
  virsh destroy ceph-iscsi$i.$dns_domain
  virsh undefine ceph-rgw$i.$dns_domain --remove-all-storage
  virsh undefine ceph-mds$i.$dns_domain --remove-all-storage
  virsh undefine ceph-iscsi$i.$dns_domain --remove-all-storage
done

# Remove ifcfg files

echo "Removing monitor ifcfg files"
for mon in `seq -w 01 03`; do
  rm $tmp_dir/ceph-mon$mon -rf
  rm $tmp_dir/ceph-mds$mon -rf
  rm $tmp_dir/ceph-iscsi$mon -rf
  rm $tmp_dir/ceph-rgw$mon -rf
done

echo "Removing OSD ifcfg files"
for t in 1 2; do
  for i in `seq -w 01 04`; do
    rm $tmp_dir/ceph-t$t-osd$i -rf
  done
done

# Remove Network files

echo "Removing ceph-presentation xml file"

rm $tmp_dir/ceph-presentation.xml -rf

echo "Removing ceph-replicaton xml file"

echo "Removing ceph networks in libvirt"

for network in ceph-presentation ceph-replication; do
  virsh net-destroy $network
  virsh net-undefine $network
done

Ceph Install⌗

Requirements⌗

This guide will use ceph-deploy so requires these steps before starting

Chrony or NTP
LVM2

Example Ansible inventory file for confirming and setting up requirements.

yaml inventory.yaml


---

ceph:
  hosts:
    grafana.ceph.lab:
      ansible_host: 10.44.20.16
    ceph-mon01.ceph.lab:
      ansible_host: 10.44.20.21
    ceph-mon02.ceph.lab:
      ansible_host: 10.44.20.22
    ceph-mon03.ceph.lab:
      ansible_host: 10.44.20.23
    ceph-t1-osd01.ceph.lab:
      ansible_host: 10.44.20.31
    ceph-t1-osd02.ceph.lab:
      ansible_host: 10.44.20.32
    ceph-t1-osd03.ceph.lab:
      ansible_host: 10.44.20.33
    ceph-t1-osd04.ceph.lab:
      ansible_host: 10.44.20.34
    ceph-t2-osd01.ceph.lab:
      ansible_host: 10.44.20.41
    ceph-t2-osd02.ceph.lab:
      ansible_host: 10.44.20.42
    ceph-t2-osd03.ceph.lab:
      ansible_host: 10.44.20.43
    ceph-t2-osd04.ceph.lab:
      ansible_host: 10.44.20.44
    ceph-rgw01.ceph.lab:
      ansible_host: 10.44.20.111
    ceph-rgw02.ceph.lab:
      ansible_host: 10.44.20.112
bastion:
  hosts:
    bastion.ceph.lab:
      ansible_host: 10.44.20.90

This playbook will do the below requirement steps. Also available on GitHub

Enable and start chronyd service

bash


ansible -i inventory.yaml all -m service -a "name=chronyd state=started enabled=true"

Confirm chrony is working

bash


ansible -i inventory.yaml all -m shell -a "chronyc tracking"

Install podman

bash


ansible -i inventory.yaml all -m package -a "name=podman state=installed"

Ensure python3 is installed

bash


ansible -i inventory.yaml all -m package -a "name=python3 state=installed"

Requirements Playbook⌗

yaml nautilus-requirements-playbook.yaml


---

- name: Ceph Lab Requirements
  hosts: all                 
  gather_facts: false                                                                                                                                                
  tasks:                                                                          
    - name: Ensure packages are installed 
      package:
        name: "{{item }}"            
        state: installed
      loop:
        - epel-release
        - chrony
        - python3
        - lvm2
        - vim
        - bash-completion
    - name: Start and enable chronyd
      service:
        name: chronyd
        state: started
        enabled: true
    - name: Build hosts file
      lineinfile: 
        dest: /etc/hosts
        regexp: '.*{{ item }}$'
        line: "{{ hostvars[item]['ansible_host'] }} {{item}} {{ hostvars[item]['inventory_hostname_short'] }}"
        state: present
      loop: "{{ groups['all'] }}"

- name: Install ceph-deploy
  hosts: bastion
  gather_facts: false
  tasks:
    - name: Add Ceph repo
      yum_repository:
        name: ceph
        description: "Ceph Nautilus repo" 
        baseurl: "https://download.ceph.com/rpm-nautilus/el7/$basearch"
        enabled: yes
        gpgcheck: yes
        gpgkey: https://download.ceph.com/keys/release.asc
        priority: "2"
    - name: Add Ceph noarch repo
      yum_repository:
        name: ceph-noarch
        description: "Ceph Nautilus noarch repo"
        baseurl: "https://download.ceph.com/rpm-nautilus/el7/noarch"
        enabled: yes
        gpgcheck: yes
        gpgkey: https://download.ceph.com/keys/release.asc
    - name: Install ceph-deploy
      package:
        name: ceph-deploy
        state: installed
    - name: Install ceph common
      package:
        name: ceph-common
        state: installed

Ceph-Deploy⌗

Following steps are run from the bastion.ceph.lab node

This section is described in detail on the Ceph Docs Site

Monitors and Managers⌗

Make new ceph directory on the bastion node

bash


mkdir ~/ceph

Change to the new ceph directroy

bash


cd ceph

Create the cluster

bash


ceph-deploy new ceph-mon01 ceph-mon02 ceph-mon03

Add the below lines to ceph.conf

bash


public_network = 10.44.20.0/24
cluster_network = 172.16.20.0/24

Install Ceph packages to the monitors

bash


ceph-deploy install --release nautilus ceph-mon01 ceph-mon02 ceph-mon03

Deploy the initial monitors

bash


ceph-deploy mon create-initial

Deploy the admin keys to the monitors

bash


ceph-deploy admin ceph-mon01 ceph-mon02 ceph-mon03

Add the MGR daemon to the first 2 monitors

bash


ceph-deploy mgr create ceph-mon01 ceph-mon02

Ensure ceph-common package is installed on the bastion and copy all config to /etc/ceph/

bash


cp /root/ceph/* /etc/ceph

Check ceph commands work from the bastion node

bash


ceph -s

Example output

bash


  cluster:
    id:     8c010781-6ede-46d2-84c8-2736c18eb382
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3
 
  services:
    mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 (age 4m)
    mgr: ceph-mon01(active, since 3m), standbys: ceph-mon02
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

OSDs⌗

Install ceph packages on all the OSD nodes

bash


ceph-deploy install --release nautilus \
  ceph-t1-osd01 \
  ceph-t1-osd02 \
  ceph-t1-osd03 \
  ceph-t1-osd04 \
  ceph-t2-osd01 \
  ceph-t2-osd02 \
  ceph-t2-osd03 \
  ceph-t2-osd04

Create OSDs on the available drives. In this example /dev/vdb -> /dev/vde are available for OSDs

bash


for n in ceph-t{1..2}-osd{01..04}; do 
  for d in {b..e}; do
    ceph-deploy osd create --data /dev/vd$d $n;
  done
done

Once this has completed, check that all OSDs have been registered in the cluster

bash


ceph -s

Example output

bash


  cluster:
    id:     8c010781-6ede-46d2-84c8-2736c18eb382
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 (age 40m)
    mgr: ceph-mon01(active, since 39m), standbys: ceph-mon02
    osd: 32 osds: 32 up (since 37s), 32 in (since 37s)
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   33 GiB used, 207 GiB / 240 GiB avail
    pgs:

Add Rados Gateway⌗

For other daemon types, Ceph docs details how to configure them.

Install Ceph on the RGW nodes

bash


ceph-deploy install --release nautilus ceph-rgw01 ceph-rgw02

Add the below lines to ceph.conf to deploy RGWs with Beast as the front end

bash


[client.rgw.ceph-rgw01]
rgw frontends = "beast port=80"

[client.rgw.ceph-rgw02]
rgw frontends = "beast port=80"

Deploy the RadosGW daemons

bash


ceph-deploy rgw create ceph-rgw01 ceph-rgw02

Now there should be 4 new pools created for the RGW objects

bash


ceph df

Example output

bash


RAW STORAGE:
    CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED 
    hdd       240 GiB     207 GiB     559 MiB       33 GiB         13.57 
    TOTAL     240 GiB     207 GiB     559 MiB       33 GiB         13.57 
 
POOLS:
    POOL                    ID     PGS     STORED      OBJECTS     USED        %USED     MAX AVAIL 
    .rgw.root                1      32     1.2 KiB           4     768 KiB         0        60 GiB 
    default.rgw.control      2      32         0 B           8         0 B         0        60 GiB 
    default.rgw.meta         3      32         0 B           0         0 B         0        60 GiB 
    default.rgw.log          4      32         0 B         175         0 B         0        60 GiB

Ceph Dashboard⌗

These steps outline how to enable the ceph dashboard

Install the Ceph Dashboard package on the monitor nodes

bash


ansible -i inventory.yaml ceph-mon* -m package -a "name=ceph-mgr-dashboard state=installed"

Enable the Dashboard module

bash


ceph mgr module enable dashboard

Disable SSL on the Dashboard

bash


ceph config set mgr mgr/dashboard/ssl false

Setup the admin user and password for the dashboard

bash


ceph dashboard ac-user-create admin password administrator

Access to the dashboard should now be availabe at http://10.44.20.21:8080 using the above crentials

Add RGW Management to Dashboard⌗

Create an admin RGW user

bash


radosgw-admin user create --uid=dashboard --display-name=dashboard --system

Grab the access and secret key for this user

bash


radosgw-admin user info --uid=dashboard

Configure the dashboard to use these keys

bash


ceph dashboard set-rgw-api-access-key <access_key>
ceph dashboard set-rgw-api-secret-key <secret_key>

Add a User to Rados Gateway⌗

Object Gateway users can either be created via the dashboard, assuming this has been configured or via the CLI

To add a user to the Object gateway

bash


radosgw-admin user create --uid=test --display-name=test

Conclusion⌗

At this point there should be a running ceph cluster at release Nautilus with, optionally 2 Rados Gateway nodes and the associated pools.

Cleanup⌗

There is a teardown.sh script in the Github repo which will remove all the VMs and their storage volumes.

Ceph Nautilus Lab

Contents

Summary⌗

Setup VMs⌗

Detailed Setup⌗

Create Networks⌗

Create VM Example⌗

Quick Setup⌗

Demo⌗

Adding More Disks⌗

Cleanup⌗

Ceph Install⌗

Requirements⌗

Requirements Playbook⌗

Ceph-Deploy⌗

Monitors and Managers⌗

OSDs⌗

Add Rados Gateway⌗

Ceph Dashboard⌗

Add RGW Management to Dashboard⌗

Add a User to Rados Gateway⌗

Conclusion⌗

Cleanup⌗