How-to delete RBD snapshot in Luminous+ Ceph Cluster (Kraken bug)
Permalink: 07/30/19, 11:07
by Konstantin Shalygin
in misc
tags: ceph kraken jewel luminous rbd snapshot
Some cluster may be affected of bug 19413:
The issue occurred when the OSDs were upgraded to Kraken and a Jewel RBD client created a snapshot. The Kraken OSD incorrectly wrote the four 0xFF values (-1) instead of four 0x00 values since the Jewel RBD client doesn't understand snapshot namespaces. When the RBD client was later upgraded to Kraken, it interpreted that namespace value as an invalid namespace and returns a -EINVAL error code.
Easy fix for Jewel compatible cluseters:
If your cluster still compatible with old clients (like Jewel), i.e. your min_compat_client is not luminous+ - just use Hammer (0.94.xx) or Jewel (10.2.xx) RBD client.
Luminous+ Clusters need manual intervention:
1. Find rbd header object.
# rbd info replicated_rbd/volume-0720f58e-69cf-4728-b6f4-1fd8bce43c14.deleted
rbd image 'volume-0720f58e-69cf-4728-b6f4-1fd8bce43c14.deleted':
size 15GiB in 3840 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.2765c3226ec1b8
format: 2
features: layering, striping, exclusive-lock, object-map, fast-diff
flags:
stripe unit: 4MiB
stripe count: 1
2. For this image rbd header is 2765c3226ec1b8, list omap values.
# rados -p replicated_rbd listomapvals rbd_header.2765c3226ec1b8
features
value (8 bytes) :
00000000 1f 00 00 00 00 00 00 00 |........|
00000008
flags
value (8 bytes) :
00000000 00 00 00 00 00 00 00 00 |........|
00000008
object_prefix
value (27 bytes) :
00000000 17 00 00 00 72 62 64 5f 64 61 74 61 2e 32 37 36 |....rbd_data.276|
00000010 35 63 33 32 32 36 65 63 31 62 38 |5c3226ec1b8|
0000001b
order
value (1 bytes) :
00000000 16 |.|
00000001
size
value (8 bytes) :
00000000 00 00 00 c0 03 00 00 00 |........|
00000008
snap_seq
value (8 bytes) :
00000000 10 00 00 00 00 00 00 00 |........|
00000008
snapshot_0000000000000010
value (149 bytes) :
00000000 06 01 8f 00 00 00 10 00 00 00 00 00 00 00 36 00 |..............6.|
00000010 00 00 76 6f 6c 75 6d 65 2d 31 30 65 31 36 63 33 |..volume-10e16c3|
00000020 30 2d 31 37 61 64 2d 34 31 61 62 2d 61 66 31 31 |0-17ad-41ab-af11|
00000030 2d 36 65 39 65 35 65 65 34 66 62 30 33 2e 63 6c |-6e9e5ee4fb03.cl|
00000040 6f 6e 65 5f 73 6e 61 70 00 00 00 c0 03 00 00 00 |one_snap........|
00000050 03 00 00 00 00 00 00 00 01 01 1c 00 00 00 ff ff |................|
00000060 ff ff ff ff ff ff 00 00 00 00 fe ff ff ff ff ff |................|
00000070 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000080 00 00 00 01 01 04 00 00 00 ff ff ff ff 00 00 00 |................|
00000090 00 00 00 00 00 |.....|
00000095
3. Backup snapshot_0000000000000010 key to file.
# rados -p replicated_rbd getomapval rbd_header.2765c3226ec1b8 snapshot_0000000000000010 /tmp/snap.orig
4. For key snapshot_0000000000000010 replace value.
FF FF FF FF's in
00000080 00 00 00 01 01 04 00 00 00 ff ff ff ff 00 00 00 |................|
to 00 00 00 00.
Convert binary to shell code with hexdump formatter:
# hexdump -v -e '"\\\\""x" 1/1 "%02x" ""' /tmp/snap.orig
\\x06\\x01\\x8f\\x00\\x00\\x00\\x10\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x36\\x00\\x00\\x00\\x76\\x6f\\x6c\\x75\\x6d\\x65\\x2d\\x31\\x30\\x65\\x31\\x36\\x63\\x33\\x30\\x2d\\x31\\x37\\x61\\x64\\x2d\\x34\\x31\\x61\\x62\\x2d\\x61\\x66\\x31\\x31\\x2d\\x36\\x65\\x39\\x65\\x35\\x65\\x65\\x34\\x66\\x62\\x30\\x33\\x2e\\x63\\x6c\\x6f\\x6e\\x65\\x5f\\x73\\x6e\\x61\\x70\\x00\\x00\\x00\\xc0\\x03\\x00\\x00\\x00\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x01\\x1c\\x00\\x00\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\xfe\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x01\\x04\\x00\\x00\\x00\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00
Replace FF bytes to 00:
# echo -en \\x06\\x01\\x8f\\x00\\x00\\x00\\x10\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x36\\x00\\x00\\x00\\x76\\x6f\\x6c\\x75\\x6d\\x65\\x2d\\x31\\x30\\x65\\x31\\x36\\x63\\x33\\x30\\x2d\\x31\\x37\\x61\\x64\\x2d\\x34\\x31\\x61\\x62\\x2d\\x61\\x66\\x31\\x31\\x2d\\x36\\x65\\x39\\x65\\x35\\x65\\x65\\x34\\x66\\x62\\x30\\x33\\x2e\\x63\\x6c\\x6f\\x6e\\x65\\x5f\\x73\\x6e\\x61\\x70\\x00\\x00\\x00\\xc0\\x03\\x00\\x00\\x00\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x01\\x1c\\x00\\x00\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\xfe\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x01\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00 | rados -p replicated_rbd setomapval rbd_header.2765c3226ec1b8 snapshot_0000000000000010
5. Lookup to this key for a result.
# rados -p replicated_rbd getomapval rbd_header.2765c3226ec1b8 snapshot_0000000000000010
value (149 bytes) :
00000000 06 01 8f 00 00 00 10 00 00 00 00 00 00 00 36 00 |..............6.|
00000010 00 00 76 6f 6c 75 6d 65 2d 31 30 65 31 36 63 33 |..volume-10e16c3|
00000020 30 2d 31 37 61 64 2d 34 31 61 62 2d 61 66 31 31 |0-17ad-41ab-af11|
00000030 2d 36 65 39 65 35 65 65 34 66 62 30 33 2e 63 6c |-6e9e5ee4fb03.cl|
00000040 6f 6e 65 5f 73 6e 61 70 00 00 00 c0 03 00 00 00 |one_snap........|
00000050 03 00 00 00 00 00 00 00 01 01 1c 00 00 00 ff ff |................|
00000060 ff ff ff ff ff ff 00 00 00 00 fe ff ff ff ff ff |................|
00000070 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000080 00 00 00 01 01 04 00 00 00 00 00 00 00 00 00 00 |................|
00000090 00 00 00 00 00 |.....|
00000095
6. Now snapshots-related operations is possible with Luminous+ clients.
# rbd snap purge replicated_rbd/volume-0720f58e-69cf-4728-b6f4-1fd8bce43c14.deleted
Removing all snapshots: 100% complete...done.