Notice

This document is for a development version of Ceph.

Design of Pool Migration

Background

The objective for pool migration is to be able to migrate all RADOS objects from one pool to another within the same cluster non-disruptively. This functionality is planned for the Umbrella release.

Use cases for pool migration:

This provides the ability to change the erasure code profile (and in particular the choice of K and M) non-disruptively. Implementing this as a non-disruptive migration between pools is simpler and no less efficient than trying to perform this type of transformation in place.
Converting between replica and erasure coded pools. Changes being made to add OMAP and class support to EC pools will remove the need to have a separate replica pool for metadata when using RBD, CephFS or RGW, this should make these migrations viable in conjunction with this work.
The general use case of wanting to migrate data between two pools.

By non-disruptive we mean that there will be no time where I/O or applications need to switch from using the old pool to the new pool, not even a very short outage at the start or end of the migration (as for example is required by RBD live migration). The migration will however need to read and write every object in the pool so there will be a performance impact during the migration - similar to that when PGs are backfilling. The same techniques and controls that are used when splitting/merging PGs and backfilling individual PGs will be used by pool migration to manage this impact.

For the first release we will require that the target pool is empty to begin with (this means we don’t need to worry about objects with the same name). See the section on avoiding name collisions for more details. Supporting merging of pools (either where constraints prevent object name collisions or where these collisions are resolved automatically during the migration) is a possible future enhancement, but it’s not clear what use cases this solves.

During a pool migration restrictions are placed on the source pool, it is not permitted to modify the number of PGs (to cause splits or merges). See sections on stopping changes to the number of PGs during migration and on the CLI and UI for more details. Deletion of the source pool will not be permitted during a migration. Other actions such as rebalancing are permitted but perhaps should be discouraged as the data is being moved anyway.

During a pool migration restrictions are placed on the target pool, it is not permitted to migrate this pool to another (i.e. no daisy chained or cyclical migrations). Splits, merges and rebalancing are permitted. Deletion of the target pool will not be permitted during a migration. Once a pool has finished migrating it is permited to start a new migration of the target pool of a previous migration.

For the first release there is no option to cancel, suspend or reverse a pool migration once it has started.

For the first release there is no plan to have the clients update their references to the pool once the migration has completed, they will continue to reference the old pool and objector (librados) will reroute the request to the new pool. The OSDMap will retain stub information for the old pool redirecting to the new pool.

The feature requires changes to client code; all clients and daemons will need to be upgraded before a pool migration is permitted. The two main clients, objector (in librados) and the kernel client will be updated. Updates to the kernel client are likely to lag the Umbrella release. Where the clients are integrated into other products (e.g. ODF) these products will need to incorporate the new clients before the feature can be used.

For the first release there is no plan to support pool migration between Ceph clusters. Theoretically this could be added later building upon the first release code but would require substantial extra effort. It would require clients to be able to update references to the cluster and pool once the migration had completed and for clients to be able to redirect I/O to a different cluster. There would be extra authentication challenges as all OSDs and clients in the source cluster would need to be able to submit requests to the target cluster.

Design

Reuse of Existing Design

Let’s start by looking at existing code or features that we can copy / reuse / refactor / take inspiration from, we don’t want to reinvent the wheel or repeat past mistakes.

Backfill

Backfill is a process run by a PG to recover objects on an OSD that has either just been added to the PG (starts with no objects) or has been absent from the PG for a while (has some objects that are up to date, some that are stale, is probably missing new objects and may have objects that are no longer needed because they were deleted while it was absent).

Backfill takes a long time so I/O must be permitted to continue while the backfill happens. It uses the fact that all objects have a hash and that it is possible to list objects in hash order. This means that backfill can recover objects in hash order and can simply keep a watermark hash value to track what progress has been made. I/Os to objects with a hash below the watermark are to an object that has been recovered and need to update all OSDs including the backfilling OSD. I/Os to objects with a hash above the watermark can ignore the backfilling OSDs as the backfill process will recover this object later. The object(s) currently being recovered by the backfill process are locked to prevent I/O for the short time it takes to backfill an object.

Another property of backfill is that the process is idempotent, while there are performance benefits to preserving the watermark there is no correctness issues if the watermark is reset to the start and the backfill process starts again as repeating the process will determine that objects have already been recovered. This simplifies the design because the watermark doesn’t have to be rigorously replicated and checkpointed, although for backfill it is part of pg_info_t so progress is checkpointed fairly frequently.

Relevance to pool migration:

Pool migration can list objects in hash order and migrate them to the new pool.
A watermark can be used to keep track of which objects have been migrated. There is no need for the watermark to be persistent.
Clients can cache a copy of the watermark to help direct I/Os to the correct pool and PG.
The client’s cached copy can become stale, if I/Os are misdirected they will be failed providing an up-to-date watermark so the I/O can be retried.
Backfill recovers all parts of a RADOS object - attributes, data and OMAP. Large objects are recovered in phases (something like 2MB at a time) and utililize a temporary object which is renamed at the end of the recovery for atomicity. If a peering cycle interrupts the process, then the temporary object is discarded. If pool migration uses this technique it needs to be aware that a peering cycle might disrupt the target pool but not the source pool and therefore may need to restart the migration of the object if the target discards the object.
Backfill recovers a RADOS head object and its associated snapshots at the same time and uses locking (e.g. PrimaryLogPG::is_degraded_or_backfilling_object and PrimaryLogPG::is_unreadable_object) to ensure that none of these can be accessed while they are being recovered because of dependencies between them. Pool migration needs to migrate the head object and the snapshots at the same time and needs to ensure we don’t process I/O to the object halfway through this process.
Backfill is meant to preserve the space-efficiency of snapshots when recovering them using the information in the snapset attribute to work out which regions of the snapshots are clones - see calc_clone_subsets / calc_head_subsets. This hasn’t been implemented for EC pools yet and tracker 72753 shows it currently isn’t working for replica pools either. We will want to use this (and the way a PushOp re-establishes cloned regions) for migration.

Unlike backfill, for migration we want the clients to know the watermark so they can route I/Os to the old/new pool. We don’t care if clients have a stale watermark - this will just cause a few I/Os to be incorrectly routed to the old pool which can fail them back to the client and communicate a new watermark so the I/O can be resubmitted to the new pool.

We deliberately make updating the client’s copy of the watermark lazy - there could be hundreds or thousands of clients so updating them all the time would be expensive. Putting the watermark into the OSDMap and issuing new epochs to distribute it to all the clients would be even more expensive. In contrast we are thinking about recording which PGs are migrating/have finished migrating in the OSDMap - a rule of thumb would be to try and only update the OSDMap once a second during a migration.

For migration to be able to support direct reads we do need all the OSDs in the PG to know where the watermark is and for this to be updated as each object is migrated. Migrating an object involves reading it from the source pool, writing it to the target pool and then deleting it from the source pool. Other OSDs can update migration progress as they process the delete request. There will be some complexity regarding direct reads and migrating an object + its snapshots. There is already some code that fails direct reads with EAGAIN (to redirect these to the primary) when an object + its snapshots have not all been recovered, we may need to use this when midway through migrating an object + snapshots and then have the primary stall the I/O until the object + snapshots have all been migrated before failing the I/O again for redirection to the new pool.

The watermark doesn’t necessarily need to be checkpointed to disk, it is cheap to find the object with the lowest hash in a PG so we could do this to recalculate the watermark whenever peering starts migration.

Scheduling Backfill / Recovery

Deciding how to prioritize backfill/recovery and how fast to run this process versus processing I/O from clients is a complex problem. Firstly, a decision is made as to which PGs should be backfilling/recovering, and which should wait. This involves messages between OSDs and considers whether I/O is blocked and how much redundancy the PG has left (for example a replica-3 pool with 2 failures is prioritized over a replica-3 pool with 1 failure). Secondly once a PG has been selected to backfill/recover the schedule has to decide how frequently to perform backfill/recovery versus process client I/O. This happens within the primary OSD using weighted costs.

Relevance to pool migration:

Pool migration is less critical than backfill or recovery. It needs to fit into the same process to determine when a PG should start migrating.
Once a PG is permitted to start migration the OSD scheduler needs to pace the work. The overheads for migrating an object are like the overheads for backfilling an object so hopefully we can just copy the backfill scheduling for migration.

The objective is to reuse as much of the scheduler (e.g. mclock) as possible, just teaching it that migration has a lower priority than backfill or async recovery but higher priority than deep scrub.

Mclock works by assigning a weighting to each backfill / recovery op and each client I/O request, it also benchmarks OSDs at startup to get some idea what the maximum performance of the OSD is. This information is then used to work out when to schedule background work. The same concepts should work for migration requests. We will need to assign a weighting to migration work; this should be similar/identical to the weighting for backfills.

We will take a similar approach for supporting clusers running with WeightedPriorityQueue scheduling.

The expectation is that there should be no need for new tuneable settings for migration, the existing tuneable settings for backfill/recovery should be sufficient, we don’t want to further complicate this part of the UI.

Statistics

I believe there are a few statistics collected about the performance of backfill/recovery. We should supplement these with similar statistics about the process of migrations.

We need to consider OSD stats that are gathered by Prometheus and any progress summary that is presented via HealthCheck and/or the UI.

CopyFrom

CopyFrom is a RADOS op that can copy the contents of an object into a new object. It is sent to the OSD and PG that will store the new object. The OSD is responsible for reading the source object which involves sending messages to another OSD and PG and then writing the data it reads to the new object. If the object being copied is large, then the copy operation is broken up into multiple stages and this is made atomic by using a temporary object to store the new data until the last data has been copied at which point the temporary object can be renamed to become the new object.

Relevance to pool migration:

Pool migration needs to copy objects from the old pool to a new pool - this will involve one OSD and PG reading the object and another OSD and PG writing the object.
Pool migration will want to drive the copy operation from the source side, so we probably need a CopyTo type operation.
The way messages are sent between OSDs, the way a large object copy is staged and the use of a temporary object name when staging are all concepts that can be reused.

Alternatively, pool migration might want to copy the recover object implementation in ECBackend which is used to recover an object being recovered or backfilled. This also stages the recovery of large objects using a temporary object and uses PushOp messages to send data to the OSDs being backfilled. It might be possible to use most of the recover object process without changes, just changing the PushOp messages to be sent to a different PG and sending the messages for all shards as the entire object is being migrated.

Lets consider the differences between the backend recovery op and CopyFrom:

CopyFrom is a process that runs in PrimaryLogPG above either the replica or ECBackend that copies an object from a primary OSD for one PG to the primary OSD for another PG. In the case of EC the primary OSD may need to issue SubOp commands to other OSDs to read/write the data.
run_recovery_op implemented by replica and EC pools runs on the primary OSD and reads data (in the case of EC issuing SubOp commands to other OSDs) but then issues PushOp commands to write the recovered data to the destination OSDs.
CopyFrom working at the PrimaryLogPG level ensures that the copied object is included in the PG stats and gets its own PG log entry so the update can be rolled forward/backwards and can be recovered by async recovery.
run_recovery_op is implemented at the PGBackend level and assumes the PG already has stats and a PG log entry for the object, it is just responsible for bringing other shards in the PG up to date.
CopyFrom ends up issuing read and write ops to the PGBackend, it doesn’t provide techniques for copying a snapshot and preserving its space-efficiency.
run_recovery_op is meant to preserve space-efficiency of clones (not implemented yet for EC pools and replica pools have bugs) – the PushOp message includes a way of describing which parts of an object should be clones.

For pool migration we probably want a hybrid implementation. We can probably re-use a lot of the run_recovery_op code to read the object that we want to migrate, and ideally handle the space-efficiency in snaps. Instead of issuing PushOps we probably want to issue a new COPY_PUT type op to the priamry PG of the target pool, but passing the same kind of information as a PushOp so we can keep track of what needs to be cloned. The target pool can then submit a mixture of write and clone ops to the PGBackend layer to create the object as well as updating the PG stats and creating a PG log entry.

Splitting PGs

Normally a pool has a number of PGs that is a power of 2. This is because we want each PG to hold roughly the same number of objects, and we use the most significant N bits of the object hash to select which PG to use. However, when doubling the number of PGs that a pool has this causes approximately half the objects in the pool to need to be moved to a new PG. We don’t want all this migration to happen at once; we want it to be paced over time to have less impact. To deal with this the MGR controls the increase in the number of PGs, it has a target for how many PGs the pool should have and slowly increases the number of PGs waiting for PGs to finish recovery before doing further splits.

When a pool has a non-power of 2 number of PGs this means that not all PGs are the same size. For example, if there are 5 PGs then PGs 0 and 4 will be half the size of PGs 1 to 3 because the choice between PG 0 and 4 is based on one extra bit of the object hash. While this is not desirable as a long-term state it is fine during the splitting process.

Relevance to pool migration:

Pool migration needs to migrate all the objects in all the PGs in the old pool to the new pool. Just like splitting we don’t want to overwhelm the system while performing the migration.
Pool migration should therefore migrate one (or a small number) of PGs at a time.
A process needs to monitor the progress of migrations, notice when PGs finish migrating and start the next PG. This could either be in the MON (in which case it would need to be event driven with OSDs telling the MON when a PG has finished migrating - somewhat similar to how PG merges work) or it could be implemented in the MGR (in which case the MGR can poll the state of the PGs and then tell the MON via a CLI command to start the next PG migration).

Direct I/O / Balanced Reads

The EC direct I/O feature is making changes to the client to decide which OSD to send client I/O requests to, it is building on top of the balanced reads flag for replica pools which tells the client to distribute read I/Os evenly across all the OSDs in a replica PG rather than sending them all to the primary.

Relevance to pool migration:

It’s changing code in the client at a similar place to where we want the client to implement pool migration deciding which pool (and hence PG and OSD) to send I/O to.
Direct I/O / balanced reads are permitted to be failed by the OSD that receives the request with EAGAIN to deal with corner cases where the OSD is unable to process the I/O. In this case the client retries the I/O but sends it to the primary OSD. A similar retry mechanism is going to be required when a client issue an I/O to the wrong pool because an object has been recently migrated. When I/Os are retried, we need to worry about ordering as this generates opportunities for I/Os to overtake or be reordered. See section Read/Write ordering below.
Direct I/O is adding extra information to the pg_pool_t structure that is part of the OSDMap that gets sent to every Ceph daemon and client by the monitor. This extra information is being used to determine that direct I/O is supported and to help work out where to route the I/O request. Pool migration will similarly need to add details to pg_pool_t structure so that clients are aware that a migration is happening.

Read/Write Ordering

Ceph has some fairly strict read / write ordering rules. Once a write has completed to the client any read must return the new data. Prior to the write completing a read is expected to return all old data or all new data (a mixture is not permitted). If writes A and B are issued concurrently one after another to the same object then write A is expected to be applied before write B – ordering of the writes is expected to be preserved through the client, messenger and the OSD. If write A and read B are issued concurrently then there is scope for read B to overtake write A. There is a flag RWORDERED that can be set that prevents this overtaking from happening.

There are no ordering guarantees when reads or writes are issued to different objects - these objects are almost certainly stored on different OSDs and even if they are on the same OSD will be processed by different threads with different locks so can easily be reordered.

There do not appear to be many uses of the RWORDERED flag, RBD and RGW do not use the flag, CephFS uses the flag in MDS RecoveryQueue (calls filer.probe which is implemented in osdc/Filer.cc) which I think is only used in some recovery scenarios.

These rules make it tricky to implement the watermark in the client and use this to decide which pool to route I/O requests to without using something equivalent to a new epoch to advance the watermark. The problem is that if the watermark is advanced without quiescing I/O it is possible that this causes requests to be reordered.

For example:

Write A issued to old pool.
Write B issued to old pool.
Write A fails with updated watermark and is retried to new pool.
Read B with RWORDERING issued to new pool.
Write B fails and needs to be retried to new pool.

In this example read B has overtaken write B.

Perhaps more concerning is that the rules would also be broken if instead of Read B we issued another write to B.

The simplest way to prevent reordering violations is to not advance the watermark while there are outstanding writes (or reads with RWORDERING flag set) in flight. This isn’t idea as it may result it quite a number of I/Os being failed for retry before the watermark can be updated.

A more sophisticated implementation stalls issuing new writes to objects with a hash between the old and new watermark while there are other writes in flight to objects with a hash between the old and new watermark.

Other Pool Migration Issues

Other topics that we need to think about for pool migration.

Avoiding Name Collisions

For the first release we will require that the target pool is empty when the migration starts (by having a UI interface that only starts a migration while a new pool is being created). We can also protect against objects being written to the target pool during the migration by adding client code to reject attempts to initiate requests to the target pool (the client code itself is still permitted to redirect requests from the source pool to the target pool). Because we will require a minimum client version to use pool migration this will ensure that all clients include this extra policing. OSDs cannot themselves implement the policing so there is no protection against a rouge client – we probably should have migration halt rather that crash if a name collision is found.

Post first release if there is a use case for merging pools then it is theoretically possible to deal with name collisions by additionally using the pool which the client is accessing the object from to uniquify the name. This would require extra information in the request from the client to the OSDs.

Stopping Changes to the Number of PGs During Migration

During a migration we don’t really want to be changing the number of PGs in the source pool. There are three reasons why:

We don’t really want to be moving objects around in the source pool when we are about to migrate them - we are probably better off getting on with the migration than trying to fix any imbalance in the source pool.
Splitting/merging PGs in the source pool makes it harder to schedule the migration. Scheduling is done at two levels - we say how many source PGs are migrating at a time and then control the rate of migration within a source PG. If we split/merge the source pool this makes selecting which PGs to migrate more difficult.
If we block splits and merges and migrate the PGs in reverse order (starting with the highest numbered PG in the pool) then we can reduce the number of PGs in the source pool as PGs finish migrating. This helps keeps the overall number of PGs more manageable.

In contrast we don’t really care so much about the target pool - we can easily cope with splits/merges while the migration is in progress. From a performance perspective we do however want to avoid migrating objects to the target pool and then having splits/merges occur that copy the objects a second time. That means that normally we would want to set the number of target pool PGs to be the same as the source pool at the start of the migrate.

We might also want to default to disable the auto-scaler for the target pool during the migration as we don’t want it seeing a nearly empty target pool with loads of PGs and thinking that it should reduce the number of PGs.

CLI and UI

Pool migration will need a new CLI to start the migration, there will also need to be a way of monitoring PGs that a migrating and the progress of the migration. The CLI to start a migration will need to be implemented by the MON (OSDMonitor.cc already implements most of the pool CLI commands) because the migration will need to update the pg_pool_t structures in the OSDMap to record details of the migration.

The new map will then be distributed to clients and OSDs so that they know that the migration has started. PGs that have been scheduled to start migration will need to determine at the end of the peering process that they don’t need to recovery or backfill and that they should attempt to schedule a migration (will need new PG states MIGRATION_WAIT and MIGRATING).

We will need to work with the dashboard team to add support for pool migration to the dashboard and to provide a REST API for starting a migration.

We will want to block some CLIs while a pool migration is taking place:

We don’t want to be able to split/merge PGs in the source pool while it is being migrated (see above).
We don’t want the target pool to become the source of another migration (no chaining migrations).

Some of these CLIs are issued by MGR, in this case we probably will need to change the MGR code to either cope with the failures and/or to detect that the pool is migrating and avoid issuing the CLIs. We probably will need both as although checking if the pool is migrating before issuing a CLI is probably more efficient, it is exposed to a race hazard where the migrate may start between the check and CLI being issued.

We need to look at how the progress of things like backfill and recovery are reported in the UI (possibly by HealthCheck?) and think about how to report the progress of a pool migration. We need to think what are the right units for reporting progress (e.g. number of objects out of total objects, number of PGs out of total PGs or just a percentage).

Backwards Compatibility / Software Upgrade

Pool migration requires code changes in the Ceph daemons (MON, OSD and possibly MGR) and to the Ceph clients that issue I/O. We can’t allow a pool migration to happen while any of these are running old code because the old code won’t understand that a pool migration is happening. Old clients won’t have any way of directing I/O to the correct pool, PG and OSD and having OSDs forward all these requests to the correct OSD would be far too expensive.

Ceph daemons and clients have a set of feature bits indicating what features they support and there are mechanisms for setting a minimum set of feature bits that are required by daemons and separately for clients. Once set this prevents down-level daemons and clients connecting to the cluster. There are also mechanisms to ensure that once a minimum level has been set that this cannot be reversed.

Pool migration will need to define a new feature bit and use the existing mechanisms for setting minimum required levels for daemons and clients. The new pool migration CLIs will need to fail an attempt to start a migration unless the minimum levels have been set.

End of Migration

When a migration completes, we will have moved all objects from pool A to pool B, however clients (e.g. RBD, CephFs, RGW, …) will still have pool A embedded in their own data structures. We don’t want to force all the clients to update their data structures to point at the new pool, so instead we will retain stub information about pool A saying that it has been migrated and that all I/O should now be submitted to pool B.

Retaining a stub pg_pool_t structure in the OSDMap is cheap - there won’t be thousands of pools and there isn’t that much data stored for the pool. We will want to ensure that the old pool has no PGs associated with it, we can do this by reducing the number of PGs it has to 0 and letting the same code that runs when PGs are merged clean up and delete the old PGs.

We need to think about the consequences of this on the UI interface. While in the code we start with pool A and create and migrate objects to pool B, from the perspective of the UI we probably want to show this as a transformation of pool A and hide the existence of pool B from the user.

An alternative implementation would just show the pool redirection in the UI, so users would see an RBD image used pool A but would then find that pool A has been migrated to pool B. This alternative implementation might be better if we plan to support merging of pools (migration to a non-empty target pool) in the future.

Walkthrough of how Pool Migration Might Work

Initiating the Pool Migration

User creates a new pool, perhaps they use a new flag --migratefrom to say they want to start a pool migration.
Starting the migration as part of pool creation means we know the pool is initially empty.
Unless the user specifies a number of PGs we can ensure that the newly created pool has the same number of PGs as the source pool. There is no requirement that the number of PGs is the same, it just avoids having to perform a migration and then perform a second copy of data as the number of PGs is adjusted to cater for the eventual number of objects in the pool.
The CLI command sets up the pg_pool_t structures in the OSDMap to indicate that a pool migration is starting. We record that pool A is being migrated to pool B, and record which PG(s) we are going to start migrating. If we are going to migrate more than one PG at a time, we probably want to specify a set of PGs (e.g. 0,1,2,3) that are being migrated. Any PG in the set is migrating. Any PG not in the set that is higher than the lowest value in the set is assumed to have completed migration, any PG not in the set that is lower than the lowest value in the set is assumed to have not started migration.
We migrate PGs in reverse order - so for example if a pool has PGs 0-15 then we will start by migrating PG 15.
MON publishes new OSDMap as a new epoch.

Client

Clients use the pg_pool_t structure in the OSDMap to work out a migration is in progress.
From the range of PGs being migrated they can work out which PGs have been migrated, which have not started migrating and which are in the process of migrating.
1. If an I/O is submitted to a PG that has been migrated the object hash and new pool is used to determine which PG and OSD to route the I/O request to.
2. If an I/O is submitted to a PG that has not started migration the object hash and old pool is used to determine which PG and OSD to route the I/O request to.
3. If an I/O is submitted to a PG that is marked as being migrated the client checks if it has a cached watermark for this PG. If it does, then it uses this to decide whether to route the request to the old or new pool. If it has no cached watermark, it guesses and sends the I/O to the old pool.
If an I/O is misrouted to the wrong pool the OSD will fail the request providing an update to the watermark. The client needs to update its cached copy of the watermark and resubmit the I/O.

OSD

OSDs use the pg_pool_t structure in the OSDMap to work out if a PG needs migrating.
At the end of peering if the PG needs migrating and is not performing backfill or recovery it sets the PG state to MIGRATION_WAIT and checks with other OSDs whether they have the resources and free capacity to start the migration.
If everything is good the PG state changes to MIGRATING, sets the watermark to 0 and the scheduler is instructed to start scheduling migration work.
Migration starts by scanning the next range of objects to be migrated creating a list of object OIDs.
Each object is then migrated, with the watermark being updated after the object has been migrated.
1. The primary reads the object and sends it to the primary of the target PG which then writes the object.
2. If the object is large this is done in stages with the target using a temporary object name which is renamed when the last data is written.
3. Once an object has been migrated it is deleted from the source pool.
Client I/O checks the object hash of the client I/O with the watermark. If the I/O is below the watermark it is failed for retry to the new pool, providing the current watermark for the client to cache.
If a PG completes a migration, then it sends a message to the MON telling it that the migration has completed.

MON

When MON gets a message from an OSD saying that a migration has completed it updates the set in the pg_pool_t to record that the PG has finished migration and that the next PG is starting migration. A new OSDMap is published as a new epoch.
Because migrations are scheduled in reverse order and objects are deleted as the migration happens, this means that as PG migrations complete that we should have empty PGs that can be deleted by simply reducing the number of PGs that the source pool has. PG migrations might not complete in the order which they are started so we might have a few empty PGs hanging around that cannot be deleted until another PG migration completes.
At the end of the migration there are no more PGs to start migrating, so the set of migrating PGs diminishes. When the set becomes empty we should have also reduced the number of PGs for the source pool to zero and at this point the migration is complete. The MON can make final updates to the pg_pool_t state to indicate the migration has finished. The pg_pool_t structure needs to be kept so that clients know to direct all I/O requests to this pool to the new pool instead.
Pools can be migrated more than once, this can result in multiple stub pg_pool_t structures being kept. We do not want to have to recurse through these stubs when I/Os are submitted, so at the end of a migration the MON should attempt to reduce these redirects to a single level.

Testing and Test Tools

The objectives of testing pool migration are:

Validate that all the objects in the source pool are migrated to the target pool and that their contents (data, attributes and OMAP) are retained.
Validate that during a migration object can be read (data, attributes, OMAP) for objects that haven’t yet been migrated, objects that have been migrated and objects in the middle of being migrated.
Validate that during a migration objects can be updated (create, delete, write, update attributes, update OMAPs) for objects that haven’t yet been migrated, objects that have been migrated and objects in the middle of being migrated.
Validating pool migration under error scenarios, including resetting and failing OSDs.
Validate that snapshots, clones are migrated and can be used during a pool migration.
Validate the UI for pool migration, including restrictions placed on the UI during the migration.
Validation of migrating multiple different pools in parallel. Validation of migration a single pool multiple times in series.
Validation of pool migration with unreadable objects (excessive medium errors plus possibly other failures that defeat the redundancy of the replica/EC pool without taking it offline).
Validation of software upgrade / compatibility for both daemons (OSD, MON, MGR) and clients.
Validation of performance impact during a migration.

Pool migration makes changes to client code, so all modified clients will need testing.

Existing tools such as ceph_test_rados are good for creating and exercising a set of objects and performing some consistency checking of objects.

A simple script is probably better for creating a large number of objects and then validating the contents of the objects. Writing a script is probably better for being able to test attributes and OMAPs as well. If the script has two phases (create objects and validate objects) then these phases can be run at different times (before, during, after pool migration) to test different aspects. The script could use a command line tool such as rados to create and validate objects, using pseudo random numbers to generate data patterns, attributes and OMAP data that could then be validated. The script would need to run many rados commands in parallel to generate a decent I/O workload. There may be scripts that already exist that can do this, it may be possible to adapt ceph_test_rados to do this.

Tools such as VDBench can test data integrity of block volumes, either creating a data set and then validating it, or can continuously create and update data keeping a journal so it can be validated at any point. However block volume tools can only test object data, not attributes or OMAPs.

A tool such as FIO is best suited for doing performance measurements.

The I/O sequence tool ceph_test_rados_io_sequence is probably not useful for testing pool migration - it specializes it testing a very small number of objects and focuses on boundary conditions within an object (e.g. EC chunk size, strip size) and data integrity.

The objective should be to use teuthology to perform most of the testing for pool migration (at a minimum 1 to 5 in the list above). It should be possible to add pool migration as an option to existing tests in the RADOS suite, extending the thrashOSD class to include the option of starting a migration.

Brought to you by the Ceph Foundation

The Ceph Documentation is a community resource funded and hosted by the non-profit Ceph Foundation. If you would like to support this and our other efforts, please consider joining now.