Data migration is a key process in any system implementation, upgrade, or consolidation. Whether replacing or upgrading server and storage equipment, consolidating websites, relocating or consolidating data centers, or integrating databases as the result of a merger or acquisition, an excellent data migration strategy is vitally important to the timeliness and ultimate success of the venture.
Data migration is typically performed using an automated process rather than human resources that can be a risk for error and are more cost-effectively deployed elsewhere. The migration is done by creating a design that maps the data from the old system to the new system in a way that relates the old data formats to the new formats and requirements. Depending on the complexity of the project, few or many phases can be involved, but at a minimum includes data extraction, where the data is read from the old system, data transformation, where the old data is transformed into a format that is compatible with the new system, and data loading, where the data is written into the new system. This is called extract, transform, load (ETL). Key to a successful outcome is the ability of the software to transform not only flat files such as Excel spreadsheets, but also complex data that comes in unstructured or hierarchical formats and requires specialized and sophisticated transformations developed specifically for this task.
Often there is also a data validation step, which is designed to test and validate the data to ensure it meets the criteria of the destination environment, as well as the input file specification. After the data is loaded into the destination system or database, a data-cleansing step is used to ensure the quality of the data, eliminate redundant or obsolete information, and match the requirements of the new system.
Here are some sobering statistics to support the importance of a good data migration strategy. Data migration typically takes 30 to 40 percent of the effort in any new application project. Thirty-eight percent of these projects exceed schedule and budget, and nearly half of that thirty-eight percent go over significantly.
This is because most people don’t realize how complex data migration is. Data migration teams often find as they delve into the project that required data is unavailable and undocumented, has unknown formats and data quality issues, and resides in many more legacy systems than originally believed.
Even more importantly, teams often discover that the data must be transformed and normalized in more complex ways than originally thought to meet requirements of the target system. For one, migrations often involve high volumes of data that challenge the computer resources of organizations. Second, by nature migrations happen across heterogeneous environments with disparate source and target data structures. This data must be transformed using complex mappings and transformations that must be specially developed, often using complicated hand-coding. Finally, data consistency must be maintained between the old and new systems. In cases where multiple applications are being migrated over different times using the same database, or when a system is gradually phased in with users, complex bi-directional synchronization must be performed between the old and new system.
Data migration is the cornerstone to any system implementation, upgrade, or consolidation and a carefully planned data migration strategy is the key to a successful project that comes in on time and on budget, and delivers a quality end product.
Centerprise Data Integrator offers impressive hierarchical data mapping capabilities for overcoming the challenges of migrating complex hierarchical structures such as XML, electronic data interchange (EDI), web services, and more. With Centerprise, organizations can reduce risks and lower project costs by using a proven data migration methodology that has been successful in hundreds of complex customer projects.
The Centerprise sophisticated ETL capabilities enable users to build an enterprise data warehouse, departmental data mart, or operations data source and easily accommodate assorted data sources or volumes.
Centerprise offers data warehouse loading functionality, including the Slowly Changing Dimension (SCD) transformation. Intuitive tools built for business users as well as developers simplify the end-to-end development, debugging, and maintenance process for intricate dataflow, transformation, and cleansing procedures. Powerful yet easy-to-use workflow environments and automation tools significantly simplify complex data retrieval processes.
Centerprise integrates data integration, data quality, and profiling features in a single, more manageable environment that facilitates the creation of data integration jobs with built-in profiling, quality measurement, and data cleansing.
Astera brings powerful data management and application integration solutions within reach of any organization. Astera's open source solutions for developing and deploying data management services like ETL, data profiling, data governance, and MDM are affordable, easy to use, and proven in demanding production environments around the world. For organizations looking to jump-start a big data initiative, Astera provides applications that accelerate data loading and other aspects of Hadoop setup by enabling developers and analysts to leverage powerful Hadoop technologies like Hadoop Hive, Pig, and Sqoop without having to write Hadoop code. Astera's ESB and data services infrastructure solutions extend proven Microsoft technologies like WCF and MSMQ to deliver affordable, flexible service enablement of distributed applications. To help enterprises improve operational performance, Astera also offers packaged solutions that support business process modeling and simulation as well as rapid development, testing, and deployment of process-oriented applications..NET, SQL Server and all Microsoft-related trademarks are the property of the Microsoft, and are used with permission.