We are committed to provide service 24 X 7

Deals, Shopping, Training, Tools

Learn how to – How To Install Ceph Storage Cluster on Ubuntu 18.04 LTS

Learn how to – How To Install Ceph Storage Cluster on Ubuntu 18.04 LTS.

In this guide, we will cover the installation and configuration of a fully functional Ceph Storage Cluster on Ubuntu 18.04 LTS server. Ceph is an open source storage solution that implements distributed object storage cluster, and provides interfaces for an object, block, and file-level storage.

Components of Ceph storage cluster

The basic components of a Ceph storage cluster

  • Monitors: A Ceph Monitor (ceph-mon) maintains maps of the cluster state, including the monitor map, manager map, the OSD map, and the CRUSH map
  • Ceph OSDs: A Ceph OSD (object storage daemon, ceph-osd)
    stores data, handles data replication, recovery, rebalancing, and
    provides some monitoring information to Ceph Monitors and Managers by
    checking other Ceph OSD Daemons for a heartbeat. At least 3 Ceph OSDs
    are normally required for redundancy and high availability.
  • MDSs: A Ceph Metadata Server (MDS, ceph-mds)
    stores metadata on behalf of the Ceph Filesystem (i.e., Ceph Block
    Devices and Ceph Object Storage do not use MDS). Ceph Metadata Servers
    allow POSIX file system users to execute basic commands (like,ls, find etc.) without placing an enormous burden on the Ceph Storage Cluster.
  • Managers: A Ceph Manager daemon (ceph-mgr)
    is responsible for keeping track of runtime metrics and the current
    state of the Ceph cluster, including storage utilization, current
    performance metrics, and system load.

Ceph Cluster on Ubuntu 18.04 Architecture

My Ceph cluster deployment is based on the following system diagram

The hostnames of each server should be set accordingly, and /etc/hosts file configured to look like below on all Ceph Cluster Nodes: rgw.example.com   rgw mon01.example.com mon01 mon02.example.com mon02 mon03.example.com mon03 osd01.example.com osd01 osd02.example.com osd02 osd03.example.com osd03

Change the hostnames and IP addresses to fit your setup. As a pre-requisite, ensure your system is updated

sudo apt update
sudo apt -y upgrade
sudo reboot

Once the systems are rebooted, you can begin Ceph cluster deployment. It is recommended to have an additional node, whose work is deploying ceph on all Ceph nodes. This node is called admin node ceph-admin.example.com ceph-admin

Prepare the Ceph Admin Node

You need to add Ceph repositories to the ceph-deploy admin node, then, install ceph-deploy package. My admin  node is on IP

Import repository key

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -

Add the Ceph repository to your system. This installation will do Ceph nautilus

echo deb https://download.ceph.com/debian-nautilus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

Update your repository and install ceph-deploy:

sudo apt update
sudo apt -y install ceph-deploy

Prepare Ceph  Nodes

The admin node must have password-less SSH access to
Ceph nodes. When ceph-deploy logs in to a Ceph node as a user, that
particular user must have passwordless sudo privileges.

Add SSH user on All Ceph Nodesrgw,osd nodes & Monitors

export USER_NAME="ceph-admin"
export USER_PASS="StrOngP@ssw0rd"
sudo useradd --create-home -s /bin/bash ${USER_NAME}
echo "${USER_NAME}:${USER_PASS}"|sudo chpasswd
echo "${USER_NAME} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/${USER_NAME}
sudo chmod 0440 /etc/sudoers.d/${USER_NAME}

Confirm that the user can run sudo without a password prompt

jmutai@osd01:~$ su - ceph-admin
ceph-admin@osd01:~$ sudo su -

Generate the SSH keys on ceph-admin node, but do not use sudo or the root user. Leave the passphrase empty:

# su - ceph-admin
$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ceph-admin/.ssh/id_rsa): 
Created directory '/home/ceph-admin/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ceph-admin/.ssh/id_rsa.
Your public key has been saved in /home/ceph-admin/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:DZZdfRS1Yo+unWAkDum7juShEF67pm7VdSkfWlsCpbA ceph-admin@ceph-admin
The key's randomart image is:
+---[RSA 2048]----+
|     .  ..  .. o=|
|      o..o .  . o|
|     E .= o  o o |
|       +.O .. +  |
| . .. .oS.*. . . |
|. o.....ooo .    |
| o.. o . . o .   |
| ...= o . . + .  |
|oooo o.+.  . o   |

$ ls /home/ceph-admin/.ssh/
config  id_rsa  id_rsa.pub  known_hosts

Configure your ~/.ssh/config

$ cat /home/ceph-admin/.ssh/config 
Host osd01
  Hostname osd01
  User ceph-admin
Host osd02
  Hostname osd02
  User ceph-admin
Host osd03
  Hostname osd03
  User ceph-admin
Host osd01
  Hostname osd01
  User ceph-admin
Host mon01
  Hostname mon01
  User ceph-admin
Host mon02
  Hostname mon02
  User ceph-admin
Host mon03
  Hostname mon03
  User ceph-admin
Host rgw
  Hostname rgw
  User ceph-admin

Copy the key to each Ceph Node ( Do this from Ceph Admin Node as the ceph-admin user)

for i in rgw mon01 mon02 mon03 osd01 osd02 osd03; do
 ssh-copy-id $i

If your remote user on all Ceph Nodes is not,ceph-admin replace it with correct username.

Deploy Ceph  Storage Cluster

Ceph is sensitive to Clock drift, so ensure you setup NTP on all Ceph Nodes, especially Monitors.

sudo apt install ntp

We’re going to create a three Ceph Node cluster with one Ceph Monitor and three Ceph OSD Daemons. Once the cluster reaches a active + clean state, expand it by adding a Metadata Server and two more Ceph Monitors

Let’s start by creating a directory on our admin node for maintaining the configuration files and keys that ceph-deploy generates for the cluster.

mkdir ceph-deploy
cd ceph-deploy

The ceph-deploy utility will output files to the current directory. Ensure you are in this directory when executing ceph-deploy.

Step 1: Initialize ceph monitor nodes

Run the following commands on your admin node from the ceph-deploy directory you created for holding your configuration details

$ ceph-deploy new mon01 mon02 mon03

Sample output

ceph-admin@ceph-admin:~/ceph-deploy$ ceph-deploy new  mon01 mon02 mon03
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy new mon01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4c0720a950>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['mon01']
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7f4c07456d70>
[ceph_deploy.cli][INFO  ]  public_network                : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[mon01][DEBUG ] connected to host: ceph-admin 
[mon01][INFO  ] Running command: ssh -CT -o BatchMode=yes mon01
[mon01][DEBUG ] connection detected need for sudo
[mon01][DEBUG ] connected to host: mon01 
[mon01][DEBUG ] detect platform information from remote host
[mon01][DEBUG ] detect machine type
[mon01][DEBUG ] find the location of an executable
[mon01][INFO  ] Running command: sudo /bin/ip link show
[mon01][INFO  ] Running command: sudo /bin/ip addr show
[mon01][DEBUG ] IP addresses found: [u'', u'', u'']
[ceph_deploy.new][DEBUG ] Resolving host mon01
[ceph_deploy.new][DEBUG ] Monitor mon01 at
[ceph_deploy.new][DEBUG ] Monitor initial members are ['mon01']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf..

Step 2: Install Ceph packages:

Install Ceph Packages on all nodes.

ceph-deploy install mon01 mon02 mon03 osd01 osd02 osd03 rgw

The ceph-deploy utility will install Ceph on each node.

Sample output

[osd03][DEBUG ] Setting up ceph-base (13.2.2-1bionic) ...
[osd03][DEBUG ] Setting up python-pecan (1.2.1-2) ...
[osd03][DEBUG ] update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode
[osd03][DEBUG ] update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode
[osd03][DEBUG ] Setting up ceph-osd (13.2.2-1bionic) ...
[osd03][DEBUG ] chown: cannot access '/var/lib/ceph/osd/*/block*': No such file or directory
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
[osd03][DEBUG ] Setting up ceph-mds (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
[osd03][DEBUG ] Setting up ceph-mon (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
[osd03][DEBUG ] Setting up ceph-mgr (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
[osd03][DEBUG ] Setting up ceph (13.2.2-1bionic) ...
[osd03][DEBUG ] Processing triggers for libc-bin (2.27-3ubuntu1) ...
[osd03][DEBUG ] Processing triggers for ureadahead (0.100.0-20) ...
[osd03][DEBUG ] Processing triggers for systemd (237-3ubuntu10) ...
[osd03][INFO  ] Running command: sudo ceph --version
[osd03][DEBUG ] ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)

Step 3: Deploy the initial monitor(s) and gather the keys:

Create initial monitors by running the command:

ceph-deploy mon create-initial

A number of keyrings will be placed in your working directory.

Deploy a manager daemon:

ceph-deploy mgr create mon01 mon02 mon03

Add a Metadata Servers:

ceph-deploy mds create mon01 mon02 mon03

Step 4: Copy Ceph Admin Key

Copy the configuration file and admin key to your admin node and your Ceph Nodes:

ceph-deploy admin mon01 mon02 mon03 osd01 osd02 osd03

Step 5: Add three OSDs

I assume you have an unused disk in each node called /dev/device. Mine look like below:

root@osd01:~# lsblk 
sda      8:0    0   32G  0 disk 
|-sda1   8:1    0  487M  0 part /boot
|-sda2   8:2    0  1.9G  0 part [SWAP]
`-sda3   8:3    0 29.6G  0 part /
vdb    252:0    0    5G  0 disk 
vdc    252:16   0    5G  0 disk 
vdd    252:32   0    5G  0 disk

I have three devices to use:

  • /dev/vda
  • dev/vdb
  • dev/vdc

Be sure that the device is not currently in use and does not contain any important data. The syntax to use is:

ceph-deploy osd create --data {device} {ceph-node}

In my case, I’ll run:

for i in vdb vdc vdd; do
  for j in osd01 osd02 osd03; do
    ceph-deploy osd create --data /dev/$i $j

Note: If you are creating an OSD on an LVM volume, the argument to --data must be volume_group/lv_name, rather than the path to the volume’s block device.

The output of the lsblkcommand should have changed

# lsblk 
NAME                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                                                                                     8:0    0   32G  0 disk 
|-sda1                                                                                                  8:1    0  487M  0 part /boot
|-sda2                                                                                                  8:2    0  1.9G  0 part [SWAP]
`-sda3                                                                                                  8:3    0 29.6G  0 part /
vdb                                                                                                   252:0    0    5G  0 disk 
`-ceph--908c8792--04e8--414f--8430--faa78e9b18eb-osd--block--275c9d8b--3825--4898--9b3b--5ea080fd7137 253:0    0    5G  0 lvm  
vdc                                                                                                   252:16   0    5G  0 disk 
`-ceph--c79a5159--3980--47e8--b649--ed0c44d32d51-osd--block--a50c2ebc--8d65--4d16--9196--6f741606b3a2 253:1    0    5G  0 lvm  
vdd                                                                                                   252:32   0    5G  0 disk 
`-ceph--594ff477--943e--49d8--8e08--addf5dbccca3-osd--block--5b71bad9--7fa8--41af--a3af--48af1219aafd 253:2    0    5G  0 lvm

Step 6: Check your cluster’s health.

Check Ceph Cluster status

root@osd02:~# sudo ceph health

root@osd02:~# sudo ceph status
    id:     5594514c-4d70-4ec8-a74e-e54ac2c1994f
    health: HEALTH_OK
    mon: 1 daemons, quorum mon01
    mgr: osd01(active)
    osd: 9 osds: 9 up, 9 in
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   9.0 GiB used, 36 GiB / 45 GiB avail

Once you have added your new Ceph Monitors, Ceph will begin
synchronizing the monitors and form a quorum. You can check the quorum
status by executing the following:

ceph quorum_status --format json-pretty

Step 7: Enable Ceph Dashboard

Enable the Ceph Dashboard module:

sudo ceph mgr module enable dashboard
sudo ceph mgr module ls

Generate self signed certificates for the dashboard:

sudo ceph dashboard create-self-signed-cert

Create a user for Dashboard:

sudo ceph dashboard ac-user-create admin 'Str0ngP@sswOrd' administrator

Enabling the Object Gateway Management Frontend:

$ sudo radosgw-admin user create --uid=admin --display-name='Ceph Admin' --system

Finally, provide the credentials to the dashboard:

sudo ceph dashboard set-rgw-api-access-key <api-access-key>
sudo ceph dashboard set-rgw-api-secret-key <api-secret-key>

If you are using a self-signed certificate in your Object Gateway setup, then you should disable certificate verification:

sudo ceph dashboard set-rgw-api-ssl-verify False

Step 8: Add Rados Gateway

To use the Ceph Object Gateway component of Ceph, you must deploy an instance of RGW. Execute the following to create a new instance of Rados Gateway:

$ ceph-deploy rgw create {gateway-node}


$ ceph-deploy rgw create rgw

By default, the RGW instance will listen on port 7480. This can be changed by editing ceph.conf on the node running the RGW as follows:

rgw frontends = civetweb port=80

Resetting your Ceph Cluster

If at any point you run into trouble and you want to start over, execute the following to purge the Ceph packages, and erase all its data and configuration:

ceph-deploy purge {ceph-node} [{ceph-node}]
ceph-deploy purgedata {ceph-node} [{ceph-node}]
ceph-deploy forgetkeys
rm ceph.*

If you execute purge, you must re-install Ceph. The last rm command removes any files that were written out by ceph-deploy locally during a previous installation.

In our next article, I’ll cover how to store and retrieve objects on Ceph and configuring Ceph Clients.

Check other similar guide:

Enable and Configure REST API Access in Ceph Object Storage

Create a Pool in Ceph Storage Cluster

How To Configure AWS S3 CLI for Ceph Object Gateway Storage

Ceph Persistent Storage for Kubernetes with Cephfs

Setup GlusterFS Storage With Heketi on CentOS 8 / CentOS 7

How to Setup S3 Compatible Object Storage Server with Minio

Setup GlusterFS Storage With Heketi on CentOS 8 / CentOS 7


Your Cart is Empty

Back To Shop