The Dark Side of Data Analytics

Despite the hype, the sad reality is that only a very small percentage of the companies embarked on the digital transformation journey have been able to successfully move their data analytics solutions into production, accelerate delivery and scale. Many are stuck and churning, finding it difficult to deliver the business value promised and justify continued investments. According to the conventional wisdom, they all do the right things: appoint a new CDO or a CAO, hire new breed of data scientists and data engineers, retrain some legacy resources or let them go, acquire new tools from the leading vendors, adopt open-source technologies, go to conferences, practice agile development methodologies and the whole shooting match. So, why it is not happening? Unbeknownst to them, they are fighting with dark forces and taking unnecessary risks. “The first dark force is that the Big Data Technology Landscape is very complex and fast evolving,” says Vijitha Kaduwela, Founder and CEO, Kavi Global. This dark force drives two fundamental risks, fast solution and people skill obsolescence. The teams are busy keeping up with the technologies and maintaining existing solutions, running out of capacity to scale and deliver new capabilities. What is the answer? Technology abstraction. Kaduwela draws an analogy, “Remember back in the days when we had a dozen flavors of operating systems. One needed an army of skilled engineers to keep the data centers running.” Then came the server abstraction technologies. The need for the skilled engineers dropped and the data centers effortlessly scaled, creating a new market for commodity servers and the cloud! The second dark force is that the data analytics stack is highly fragmented. One needs to integrate at least half a dozen of technologies often from multiple vendors to implement an end-to-end solution. A tool for data ingestion, another for data integration, one for data quality, few for reporting and BI, several others for data science, machine learning, etc., Most of these technologies are not mature and require heavy duty code writing that leads to fragmentation, low productivity and the need for deep skills in delivery teams. So, what is the answer? Horizontally integrate the stack at metadata level and simplify the interface and improve the productivity of the workforce.