Big Data Cloud Migration
Business critical Hadoop cluster migration to AWS
Download Case Study
A global player in the travel industry faced a challenge for its Market Analytics product, a platform for business intelligence and analytics for airlines. Their platform had to be migrated to the cloud to increase reliability and flexibility, while customers on the production environment should not experiencing any disruption to the service. Neos IT successfully migrated the solution to AWS for typical cloud benefits with no downtime during transition.
Initial Situation
The challenge faced by the client was clearly framed from the start. The highly critical Market Insights Analytics suite needed modernization without exposing their customers to downtimes. This meant that all analytical activities for bookings and passenger data had to continue functioning during the modernization. The analytics supporting smooth decision-making processes of their customers should not be compromised at any time.
Challenges
Before modernization by Neos IT, the client’s Market Insights Analytics suite ran on physical hardware. This required intensive maintenance as well as frequent hardware upgrades and replacement, for which the client had to bear the business risk and pay in advance.
Frequent service interruptions were caused by planned maintenance, capacity increase and outages. Servers were rebooted several times, causing downtimes for customers, increased operational effort as well as costly reprocessing of data. In response, it was quickly decided that a flexible, cloud-based solution was needed. The goal was to improve availability of the platform while increasing flexibility to grow with demand, accommodate processing peaks and disconnect compute investments from ever-growing data storage.
After proposing a solution architecture, Neos IT provided a proof of concept which ultimately led to the decision to migrate the entire solution to the cloud.
Solving the problem
In the initial scope, a deep refactoring for cloud- native services was discarded in favor of a quick implementation. The cloud solution in AWS used S3 storage as well as EC2 instances to run Hadoop and Cloudera. Decoupling storage from computing allowed the customer to consume the right amount of compute power needed to process monthly, weekly or unplanned analytics jobs. Previously, an increase in compute power required the purchase of related storage, leading to costly excess capacity, lack of compute power during peaks, or both. Another benefit of S3 compared to previous storage, was high availability and durability of data. This removed the need to store data in multiple copies to meet backup and availability requirements. Compute power, on the other side, could then be consumed „in more flexible terms, like instance reservation for baseline workloads, on-demand for processing peaks and spot instances cost efficiency.
IT security and compliance procedures also had to be updated. In the field of analytics and especially when personally identifiable data is being processed, every effort must be made to prevent gaps in security. In collaboration with the client’s security team, Neos IT set up seamless, end-to-end encryption of all data with the necessary security levels.
Auch die IT-Sicherheits- und Compliance-Verfahren mussten aktualisiert werden. Im Bereich der Analytik und insbesondere bei der Verarbeitung personenbezogener Daten müssen alle Anstrengungen unternommen werden, um Sicherheitslücken zu vermeiden. In Zusammenarbeit mit dem Sicherheitsteam des Kunden richtete Neos IT eine nahtlose End-to-End-Verschlüsselung aller Daten mit den erforderlichen Sicherheitsstufen ein.
Customer value:
The technical benefits of the new solution are the ability to scale its services flexibly in the cloud, while their customers benefit from dramatically increased performance. Time-consuming routine tasks have been reduced, availability increased to 99.9% and the time required to re-scale computing capacity of the platform has been reduced from around six weeks to just four hours.
The business benefits for the client is reducing time-to-market of new features and delivering customer-specific queries much quicker. For end users this means that the Market Insight Analytics suite has higher availability and business insights can be generated quicker to react to a changing market.