Overview

The leading stock exchange in Asia recognized the imperative need for a digital transformation initiative aimed at modernizing its Data platform. The vision was to deliver an end-to-end suite of critical services that would optimize costs, improve the digital footprint, and enhance business analytics, performance, reliability, and scalability.

Challenges

• Diverse customer base:

  The stock exchange found itself grappling with a highly varied and expansive customer base that ranged from individual retail traders to large institutional investors. This multifaceted customer landscape complicated the task of data integration, making it a herculean effort. The existing Data and Analytics platform was not initially designed for such scalable needs, leading to significant challenges in accommodating this growth. Additionally, the sensitive nature of financial data added another layer of complexity, as the organization had to meet stringent data security and compliance standards to safeguard customer information.

• Insufficient knowledge and documentation:

  Over the years, the organization had accumulated a range of legacy applications with little to no documentation. This lack of comprehensive records escalated the level of operational risk, contributing to a growing pile of technical debt. The absence of well-documented processes and functionalities made it difficult to evaluate the applications’ interdependencies, and it raised the specter of potential non-compliance with regulatory standards.

• Legacy technologies:

  The organization’s existing data platform was a mosaic of outdated technologies, including Oracle and Informatica, that were becoming increasingly incompatible with modern, cloud-based solutions. These legacy systems were not only difficult to maintain but also acted as performance bottlenecks. Their limitations in scalability and interoperability added to the overall operational inefficiency.

• Migration needs:

  The stock exchange had a critical requirement to modernize by migrating its data from an ON-PREM Greenplum DB to an ON-PREM Oracle platform. This migration was fraught with challenges, as it necessitated meticulous planning to ensure data consistency and integrity. The downtime during this transition had to be minimized to avoid disrupting the market operations. Furthermore, all of these had to be accomplished within the constraints of a limited budget, making cost-effective execution a paramount concern.

Solution Approach

End-to-end data modernization:

DataSwitch initiated the modernization process with a comprehensive audit of the existing data infrastructure. This allowed them to develop a finely detailed migration strategy that covered every aspect of the data platform. They offered an integrated solution that was custom-tailored to meet the stock exchange’s unique needs and challenges.

ETL code conversion:

To address the complexities of the existing ETL processes, DataSwitch deployed their proprietary DS Migrate Process Converter. This tool facilitated the conversion of the existing Informatica ETL codes into Snowflake SQL. Beyond simple conversion, the tool also optimized the code for better performance, and rigorous testing was performed to ensure that the converted codes met all functional and performance requirements.

Automated script conversion:

DataSwitch’s automation capabilities were leveraged to expedite the conversion of complex Oracle scripts into their Greenplum equivalents. The process included automated validation checks that ensured the converted scripts were not only functionally equivalent but also maintained the integrity of the data.

Professional services:

The DataSwitch ProServ team was instrumental in the project’s success. They collaborated closely with the client’s DEV team throughout the migration process. Through technical workshops and regular status updates, they ensured that all stakeholders were aligned and informed. Their expertise and guidance were critical in overcoming the technical and operational challenges encountered during the migration.

Platform support:

Post-migration, DataSwitch provided continuous support to address any emerging issues. Their platform support included round-the-clock assistance and the deployment of advanced monitoring tools, ensuring the ongoing optimal performance of the newly modernized system.

Value Delivered

• 1600+ Informatica ETL Mappings: Successfully converted to Snowflake Scripts.
• 8000+ Table Objects: Migrated from Greenplum to Oracle, dealing with 8 Peta Bytes of data.
• 90% Automation: Achieved high automation efficiency.
• 1000+ Scripts: Converted medium to complex Greenplum Scripts to Oracle and vice versa.
• Coding Guidelines: Defined and applied coding guidelines for the conversion process.

Fast Facts

Technologies:

DS Migrate, Informatica, Greenplum, Shell Scripts, Oracle, Snowflake, Snow SQL, Apache Airflow

Engagement Model:

Platform + Professional Services

Other Details:

Enabled 20+ users with Self-Service translation services

Conclusion

DataSwitch’s comprehensive end-to-end solution empowered the leading stock exchange in Asia to successfully modernize its data platform. The approach not only saved time and costs but also dramatically improved the overall efficiency of data processing and business analytics. With a newly agile and modern data platform, the organization is now better positioned to achieve its strategic growth objectives and seize new business opportunities.