Planning a data migration in the next year or two? Is it a small or large migration? Are you having difficulty determining the scope? Now is the time to get started in the process. Follow these basic steps to a successful data migration.
Step 1 – Planning the scope of the migration:
Before the project is given a pricing and schedule estimate, we need define the scope of the migration. We begin with the basics like how many databases are we migrating data out of and how many are we putting data into. Even in cases where we are migrating from one database to another, there are important considerations to take into account:
- Do we have a complete definition of both databases? We are going to need this to complete the initial planning.
- Are we migrating all of the data?
- How much data is there to migrate? During this process it may be determined that some of the data tables will not be needed and should not be migrated.
- Also, maybe this is an appropriate time to archive historical data that is no longer needed in the daily database access. If we are archiving the old data, can the data remain in the original database after the migration or does it need to be archived elsewhere. Do we have tools in place for archiving or do we need to migrate archived data to the archive?
Answers to these questions will have a very large impact on the overall scope of your migration.
Step 2 – Data security:
Do we have a clearly defined security policy for the data? If not, is one needed? If confidential data needs to be secured during your migration, this should be taken into account at the very beginning of the project. We will need to define how data is secured in the source database as well as in the destination and also the process of moving the data. Another consideration is testing. Will confidential data be used during the test migrations? If so, we need to plan accordingly and validate that the test process is properly secured as well.
Step 3 -Define key players:
Technical resources will help ensure that data is accessible and also secure. Business users who understand the data will help with the data mapping process and to validate success during our test migrations. And finally, management needs to buy off on the plan and make sure that resources (staff) are available to work on the migration.
Step 4 – Mapping the data:
This is where we get into the details of what goes where. Data fields from one database to another frequently do not fit neatly together in a one to one mapping. There are many cases where data fields are mapped in a one to many fashion or we have to deal with situations where data from two or more fields only have one recipient field. While there are going to be many fields with a clear mapping from old to new, there are going to be much less obvious decisions to be made.
Step 5 – Scheduling the migration:
How long will the data migration take? Will there be enough time to complete the whole migration in the desired time window or will the data migration have to be completed in portions over a longer period of time?
Step 6 – Preparing the tools:
At Streamwrite, we use our own, in-house-developed, data-migration tool. Once the mappings are complete, we apply the mappings to the tool and run some basic tests to ensure that the Streamwrite Data Migration Tool can connect both databases in the test environment properly.
Step 7 – Test migrations and data validation:
Time must be allocated for testing the migration and validating the success or failure of the test migrations. Placing emphasis on this step will ensure success of our final migration. This also gives us the opportunity to verify the length of time the migration will take so that we are certain that the migration will complete in the time window allocated (overnight, over a weekend).
Step 8 – Final preparations and migration:
We will need to verify that proper permissions are applied to production databases and that there are no connectivity issues. We monitor the process during the final migrations to stay on top of any problems that might crop up such as a high number of rejected records. We have some quality control measures here to validate the final data and produce a completion report that will show the total number of records migrated and any errors that were encountered.
The proper data integration partner should be brought in early in the process. That partner can assist with the initial project scope and guide you through the process from beginning to end. It is normal for a data migration to involve hundreds or thousands of tables and millions of records. Starting the planning early with a trusted partner is the best way to ensure the successful migration of your data while maintaining required data security provisions.
Contact us if you would like more information.