This blog post touches on the broader themes of our whitepaper, Reducing the Runaway Costs of a Hybrid Big Data Setup.
The rise of the cloud has changed the face of big data. Whether through lift-and-shift or re-architecting, almost every large enterprise has carried out some migration to the cloud.
Granted, the majority of these large enterprises have kept some of their workload on-premises due to privacy concerns, ongoing licenses, and such, while others landed an organization with a large cloud profile through company acquisition. In any case, these enterprises are now managing a hybrid—and usually multi-cloud—big data environment.
The problem? The management and optimization of big data in a hybrid cloud architecture is a new and evolving challenge. As Gartner put it:
“Changes in architectures, and specifically with funding moving away from the traditional data center with the advent of cloud, hybrid cloud, virtualization, SDN/SD WAN and disaggregation, change how infrastructure monitoring needs to be conducted.”
Not only that, IT operations are now in a visibility crisis. Compared to their legacy stance, they suddenly cannot understand what they are spending or why.
“Through 2020, 80% of organizations will overshoot their cloud IaaS budgets due to a lack of cost optimization approaches” (Gartner, 2019).
Even worse, this visibility crisis is translating into a cost crisis. When enterprise IT organizations receive their first few cloud bills, many are shocked. Cloud invoices can add up to hundreds of thousands more dollars than expected.
Case in point: when Bain & Company asked more than 350 IT decision-makers what aspects of their cloud deployment had been the most disappointing, the top complaint was that the cost of ownership had either remained the same or increased.
Where does this problem begin? Why do companies overshoot their budgets? And ultimately, what should IT Operations personnel do to keep this visibility-and-cost crisis from getting worse?