Blue chip professional skills from an independent

Data migration

Data migration is generally applicable when a new system is implemented; you need to import data from the 'old, replaced' system into the new one. Occasionally, you need to bulk load data into an existing system, possibly due to a company merger or similar.


In any event there tend to be similarities between high volume interfacing and data migration. How much similarity depends on the checks and balances needed, any data mapping, data quality, how strong the audit trail needs to be, the amount of data to be processed and the time frame available in which to do it.


Summary of considerations


Checks and balances


In general, for any migrated data you will need reports and so on to compare the data in the old system to the new system. You need to know that the data has been migrated cleanly. For a finance system, a trial balance is an obvious starting point.


Data mapping and translation


It is often the case that the data in the new system will be slightly or significantly different from the old system. You may have a new set of nominals, customer and supplier codes may be changed and so on. The more complex the requirements here, the more complex the migration becomes; at its most complex it becomes a full blown interface in itself, albeit one that is used once, and then discarded.


Data quality and cleansing


It is often the case that during a migration you find that your old system had data issues that you never knew about. In the context of the migration decisions will be needed on how to correct the data so that it arrives in the new system in a 'good' state. Data cleansing is an important part of a migration. It may become a significant part of the overall project if there is much to do; or there may be a handful of cases which are trivial - they can be corrected manually in five minutes.


Audit trail


Sometimes a strong audit trail is needed, such that every piece of data in the new system can be tracked back to the corresponding piece of data in the old system. If there is much data translation and mapping, or data cleansing issues this can become quite onerous. It is possible data volumes will be such that it is just not feasible to have the audit trail you might wish for.


Data volumes


Even with the computing horsepower available nowadays the data volumes in a migration can cause considerable problems. Issues can be two fold: there is so much data that there will not be enough time to migrate it; or there is so much data you don't actually want all of it in the new system as it (for example) would slow down reports too much.


There are sensible strategies to adopt depending on your needs and restrictions. You might archive prior years' data and only bring over year end balances. You may bring data over period by period, and possibly in summarised form. The needs here will play a major part in deciding the approach to the migration.


Time frame


Generally a migration will be scheduled over a weekend. In some cases there is a small enough quantity of data it literally only takes an hour or so. In other cases, a 'full' migration, with data cleansing and lots of data translations can take several days. In these cases compromises are very attractive and usually have side benefits in any case, such as improved reporting speed.


Which approach to take?


You will see from the above there are many things that influence the approach to take.


The list that follows is a quick reference for the types of approach we have seen and might be applicable in your situation. You may end up using a mixture of these:

  • write a full blown interface using full cycle software development;
  • use a database vendor's toolset to read and manipulate data from one system to another;
  • summarise the data before inserting into the new system;
  • use spreadsheets to manipulate the data;
  • directly insert the data into the new system's database, rather than use the formal data import utilities;
  • phase the migration; bring over the crucial current data for day one, then the historic reporting data gradually over the following few days;
  • store some of the historic data elsewhere (maybe left in the old system) and run reports on it whenever the need arises.

How we can help


The first step would be for some high level analysis to understand which broad approach to take. The broad headings for the scoping would be as above, though they may move around and some new ones be needed for any given situation. From this would come a recommended approach. Once agreed we would produce a full scoping document with a project plan, actions, risks and responsibilities.


We have the skills and know-how to implement the solutions mentioned - or to assist your staff in implementing them. In other words, we can provide a turn key solution or work as part of your team. As always, pleasecontact us to discuss your specific situation. We would be more than happy to spend time with you outlining the options as we see them.