The Company

The client is a European energy company that is active in the production, trading, development and distribution of energy, renewable power sources and commodities with businesses and branches spread across the whole of Europe.

The Challenge

The company envisioned a multi-step process of upgrading their existing legacy systems and migrating their larger infrastructure towards a service-oriented cloud architecture. The first part consisted of migrating their core on premise warehouse solution to Snowflake. Then surrounding the Snowflake warehouse solution with an AWS services-based data lake and orchestration platform and migrating secondary systems and other components into the defined architecture all the while maintaining the developed principles and architecture.

The Solution

The core part of the system was the existing on premise Oracle Exadata warehouse. This was a daily processing solution that had to meet strict processing overnight requirements and provide accurate data for daily operations by the start of each business day. The solution, although usable, was starting to put pressure on the hardware infrastructure but also on business operation as data volumes increased and scalability was an obvious issue.

The solution agreed upon was to move to Snowflake as it offered a scalable cost-effective solution, high processing power and flexibility and ease convertion of the existing processes from Oracle. Once Snowflake was chosen, a new approach to processing data was also possible. The process would no longer be a daily operation, but data ingested would be processed as soon as possible in a near real time fashion. This would allow better performance, ability to integrate new systems in the architecture and to also enrich the warehouse with data from new systems.

The new architecture would also allow for revisiting the existing codebase and to convert and improve existing legacy transformations and to improve the performance by taking advantage of the new architecture and processing philosophy.

To complement the new Snowflake architecture an orchestration and ingestion pipeline was set up using AWS services. This allowed the development of an event driven services-oriented orchestration platform that has the role of streamlining data into Snowflake from other systems.

The move towards AWS allowed the setup of a multi-account DevOps oriented system architecture that improved code and processes integration and facilitated the future integration of complementary systems.

The current and ongoing stage of the project is the expansion of the previously set-up architecture with new systems that sit on top or beside the core warehouse. These extensions, due to the generic services-oriented architecture developed are easy to not only integrate with the existing processes but also flexible enough to be entirely redesigned using new technologies and standards in order to take advantage of new advances like GenAI or new AWS services and Snowflake scaling features.

During the entire project Quark was part of a large international team in charge with this migration.

The Business Impact

The implementation of the new platform allowed changes in the day-to-day operations, providing not only a reduction of the costs but also flexibility in the day-to-day business processes as well as the option of resource scalability for peak times.
The modernized service and event driven architecture allowed for easy integration of complementary systems and a flexible approach to the expansion of the business use cases.
The main goal of improving performance and reliability of the operations was achieved not only by choosing the best technologies for the use case but also by changing the architecture and the legacy approach to meet the new standards and best practices of a cloud based, services oriented, event driven, dev-ops run platform.