As the title suggests, this article focuses on the administration of various services inside a Ceph cluster.
A Ceph file system requires two pools to start with. One pool contains the data while another pool is meant for the metadata. During the installation these pools should already be created:
ceph osd lspools
11 data,12 metadata,
If this is not the case, create the pools:
ceph osd pool create data 128
ceph osd pool create metadata 128
With these pools available, a file system can be created. First make sure that no file system already exists:
ceph fs ls
name: cephfs, metadata pool: metadata, data pools: [data ]
If it already exists, then no action needs to be undertaken anymore. Otherwise the file system can be created:
ceph fs new cephfs metadata data
To remove a file system, it is necessary to first fail the MDS service:
ceph mds fail 0
ceph fs rm cephfs
When a file system exists, it can be mounted on the Linux clients that participate in the cluster. With the Cephx authentication it is necessary to pass on the client name (the client.admin one which is created when the cluster is created can be used, but a less privileged one can be used as well) with the key (usually by referring to the secret file created when creating the user key):
mount -t ceph -o name=admin,secretfile=/etc/ceph/ceph.client.admin.secret host1,host2:/ /srv/ceph
Pools can be created and manipulated immediately. However, when removing a pool, make sure that the pool is not used as all data is irrevocably removed from the cluster.
To list the current set of pools, use ceph osd lspools:
ceph osd lspools
11 data,12 metadata,13 userdata,
To get some more information, such as the number of objects, size, etc. use the rados command:
pool name category KB objects clones degraded unfound rd rd KB wr wr KB data - 0 0 0 0 0 0 0 0 0 metadata - 2 20 0 0 0 0 0 31 8 userdata - 109702 1 0 0 0 0 0 27 109702 total used 14322004 21 total avail 83099780 total space 102683912
To create a pool:
ceph osd pool create s3data 256
To remove it:
ceph osd pool delete s3data
Administrators can take a snapshot of an existing pool (for instance to simplify backup operations so that the backup can take its time, being certain that the (snapshot) pool remains constant).
ceph osd pool mksnap s3data s3data-20150701
To remove the snapshot again, use rmsnap.
To get placement group information, use ceph pg dump:
ceph pg dump
dumped all in format plain version 570 stamp 2015-07-12 16:50:43.244414 last_osdmap_epoch 147 last_pg_scan 146 full_ratio 0.95 nearfull_ratio 0.85 pg_stat objects mip degr misp unf bytes log disklog state state_stamp v reported up up_primary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp 13.72 0 0 0 0 0 0 0 0 active+clean 2015-07-12 16:42:29.120819 0'0 147:9 [0,1] 0 [0,1] 0 0'0 2015-07-12 16:42:21.359576 0'0 2015-07-12 16:42:21.359576 13.73 0 0 0 0 0 0 0 0 active+clean 2015-07-12 16:42:29.122274 0'0 147:9 [0,1] 0 [0,1] 0 0'0 2015-07-12 16:42:21.383445 0'0 2015-07-12 16:42:21.383445 ...
The output is reduced. In reality a line is shown for every placement group, which can become quite large in larger deployments.
To get the placement information for a particular placement group, use ceph pg map:
ceph pg map 13.17
osdmap e147 pg 13.17 (13.17) -> up [0,1] acting [0,1]
Authentication keys are best managed through ceph auth. During the key generation, the capabilities are added so that the cluster knows which operations are supported through the key and which ones aren't.
For instance, to create a key for a user (say client.chris) which can manipulate the s3data pool:
ceph auth get-or-create client.chris mon 'allow r' osd 'allow rwx pool=s3data' > /home/chris/client.chris.keyring
The user can use this keyring for the cluster operations. Note that the user still requires read access on the Ceph configuration file, but does not need access (and in fact shouldn't) to the secret files or keyrings inside /etc/ceph.
rados -n client.chris --keyring /home/chris/client.chris.keyring -p s3data put "objectname" /path/to/file
To list the current authentication keys:
ceph auth list
To remove all privileges from a key:
ceph auth del client.chris