We all know data Cloud costs are an increasing concern for data managers. With pricing models based on consumption data costs can, seemingly in the blink of an eye, skyrocket out of control. So, tracking these costs every now and again is simply not an option. I know through personal experience that unchecked data spikes can wreak havoc in a matter of hours.
Being caught off guard by massive jumps in costs has happened to me so many times. Usually this was because my data team changed something in the DBT model and did not realize the impact it would have. Only when we saw the spend increasing by a factor of 10 would we realize the mistake.
Let’s imagine we had a tool that could monitor data consumption allowing us to see any anomalies or changes immediately. We could spot increases in costs instantly. The problem is, without such a tool, sudden spikes in costs are only spotted a few weeks after the change that caused it has occurred. Until then more and more of the data budget will be silently eaten away.
Finding the Right Owner and Root Cause of Data Spikes
When a spike in costs is detected, the next step is to immediately find the root cause. And this can be a harder task than many people would think. Did someone change the model? Or do we have more data in the sources we are scanning? If this data increases dramatically the costs will rise because of the time it takes to process it.
As the head of data in my organization, I may have seen the increase in costs but I won’t know who caused it. Because of distributed ownership within data teams, it is not only data engineers that can influence unexpected changes in data costs. Such spikes can also be caused by a data analyst, data scientist or even a data consumer that runs an inefficient query.
Creating a Culture of Continuous Cost Vigilance
Engineers are traditionally not attuned to the financial impact of their work. The lack of appropriate tools and systems for monitoring and adjusting in response to cost variations has historically hindered this accountability. But they must embrace cost management as part of their operational mindset. So transitioning to a culture of continuous cost vigilance is paramount.
My most striking lesson came from a job that escalated from a weekly cost of $100 to an astounding $4,000 — a 40-fold increase that went unnoticed for a week due to inadequate monitoring systems. This experience underlines the necessity for data managers to have access to sensitive, accurate monitoring tools that track both quality and costs in real-time.
As organizations grow more data-dependent, the imperative for cost efficiency is becoming increasingly pronounced. To meet this challenge it is now essential for data managers to deploy a comprehensive solution that routinely monitors costs on a daily basis and uses root-cause analysis to rapidly identify the source of cost anomalies.
Deploying such a solution will pave the way for a more efficient and effective approach to managing cost spikes at a time when data usage is now an ever-increasing and significant cost for organizations.
Building Data Pipelines? Stop and Assess
Building data pipelines serves as a prime example of the intricate balance between operational functionality and cost efficiency. The frequency of data refreshes, often driven by departmental demands, when creating dashboards directly influences overall costs. Effectively managing these costs requires taking the time to contemplate what data is required and how best to deliver it.
Creativity is always a good option. I remember one specific cost optimization project we undertook for a Seemore Data client where we understood the impact data refreshes were having on costs. So we decided to turn off the pipelines between 10pm to 6am. Why? We realized that people were not using these dashboards during the night. This was only possible because of the data lineage feature in Seemore Data, which understands all the dependencies between the data products and pipelines. This means we could actually turn off the pipelines safely. Just by implementing this small change we saved a third of the money.
This narrative of cost optimization is not just about saving money, however; it’s about fostering a deeper understanding of cost implications across all levels of an organization. Imagine a typical scenario — the Marketing Director wants a new dashboard asap. We prioritize its creation and delivery. Once completed we then discover that the marketing department will only use this data once every quarter for a low-level report. This request should never have been prioritized and taken up so much budget to develop if its value and frequency-of-use were taken into account.
The cost implications of data products and their impact must be transparent across all C-Level executives. This will enable more informed and more strategic decisions around data usage to prioritize data products that add most value.
Seemore Data: Delivering Context, Alerts, Ownership and Transparency
While optimizing costs remains a challenging endeavor, it’s now critical in the data cloud era. This is why we started Seemore Data. We want to help data leaders, their teams and their organizations to benefit from:
- Context — Full visibility of the cost related to the DBT model, consumer and workflow.
- Alerts — Customized notifications to continuously and automatically monitor costs as part of a daily routine that ensures unnecessary costs are proactively identified, dramatically reducing the time it takes to fix the problem.
- Ownership — Identification of the owner so they can be directly notified of a cost spike without the need to involve the head of data.
- Transparency: The cost and impact (value) of data products to enable more informed strategic decision-making.
As we move forward, our goal at Seemore Data is to give organizations the technology and the tools to empower them to monitor their data costs more rigorously. This will not only bring far more data-cost efficiency, it will also form the bedrock on which a culture of transparency, shared responsibility and ownership of data costs becomes inherent within organizations.
This cultural transformation will redefine how organizations approach data management, enabling them to marry operational excellence with financial prudence to drive strategic value.
Learn how easy it is to gain full transparency around the cost of your data using Seemore Data , and finally put an end to the damage unforeseen data spikes can have on your budget — book a demo today.
