Big Data – A costly affair?


Share This Post

Share on facebook
Share on linkedin
Share on twitter
Share on email

Business Intelligence – Then and Now

Howard Dresner, President of Dresner Advisory is partially known for readapting the term “Business Intelligence” to “describe making better business decisions through searching, gathering, and analyzing the accumulated data saved by an organization.” He adapted that umbrella term in 1989; it can easily be said that data along with business intelligence has come a long way since then.

In the 1980’s we saw the growth of data warehouses, this was the first sign of the breadth in which data was headed. As technology progressed (Moore’s law) and hard drives consequentially got cheaper, so did data storage. The architecture of data warehouses was developed to help transform data coming from operational systems into decision-making support systems. For example, enterprises were now able to store sales data along with the respective time through a data warehouse, which enabled them to plot and analyze data in order to make decisions.


The chart above published by Mckinsey&Co shows the level of change in multiple industries’ core business practices brought on by data and analytics for the past three years. Published Jan. 2018


Following the onslaught of data warehouses, the 1990s were the era of data mining. Businesses were now able to house much more data, and technology allowed this to be done faster and more efficiently. Data mining focused on discovering patterns in large data sets, and this enabled companies to do predictive analysis through historical patterns. Businesses now began focusing on pattern recognition in data in order to make more educated forecasting decisions.

By 2005, the data collected on average was so large that business intelligence tools simply could not cope with the volume – requiring the invention of new data analysis technologies such as Hadoop. Accordingly, the term “big data” was coined by Roger Magoulas in 2005 to describe data of this size.

Data-Driven Initiatives – What do businesses want from their data?

According to a data analytics survey by IDG, the top three challenges organizations hope to solve through data-driven initiatives are: finding correlations across disparate data sources (60%), predicting customer behaviour (47%), and predict product or service sales (42%)2. Businesses are looking to make data-driven investments in order to bring further insight into their operations and segments, helping them predict trends. Businesses can take a few avenues in order to reach these goals.


The analysis of big data today is reaching a point of inefficiency from a personnel standpoint. You can always hire more analysts but the marginal return of every analyst you add to your team will decrease with time and will open you up to a host of personnel related issues. Entry-level data analysts have a yearly national average salary of $40K3, after a certain point, scalability is no longer viable through manpower. The insight deduced from the data will not justify the cost.


Pioneers in the technology industry such as Netflix and Twitter have decided to tackle the big data analysis issue by developing their own AI-powered Anomaly Detection and Automated Analytics platform. The investment made by these tech giants shows us a few things. Firstly, if Netflix and Twitter are both willing to invest the time and money it requires to develop an automated analytics tool, then clearly they see value in this tool. Secondly, Netflix and Twitter have $111B and $21B dollar valuations, making the in-house investment for anomaly detection plausible and fundable.  Thirdly, they are best practice creators, their investment into automated analytics shows where the industry is headed in terms of data management. But is automated analytics just a game for giants?

Automated Analytics

Data analysis is forcibly moving toward automated analysis. Automated analysis is done through machine learning; the software constantly scales through data to find insights that are important to your business. A business that aims to reach their data-driven initiative goals (mentioned above) will find that automated analytics through Software as a Service (SaaS) providers is the quickest route to take. Automated analysis tools through SaaS providers are quickly scalable and can generate insights more quickly at a fraction of the cost. As shown above, we saw that the in-house coding of these tools may be too costly for SMBs.

Looking back at the growth of data and accordingly the expenditure on data-driven initiatives it is fair to say big data is here to stay!

What Millimetric Offers

Millimetric is a SaaS automated analytics provider that offers you what the tech giants are

developing in-house and what you can’t scale with manpower. We provide you with:

  • Anomaly detection email alerts
  • Root cause analysis for technical issues
  • 24H Anomaly monitoring
  • Anomaly Reports and analysis tools
  • Anomaly Share & Collaborate

Don’t take our word for it, though, give it a shot and let us know what you think!


Get Started with Millimetric Today!

More To Explore

Ready to get started?

  • Try it Free
  • No credit card required
  • No code required