Logo

The Data Daily

17 Data Management Predictions from 12 Experts for 2021

17 Data Management Predictions from 12 Experts for 2021

We polled 12 experts and received 17 data management predictions for 2021.

As part of the second annual Solutions Review #BIInsightJam, we called for the industry’s best and brightest to share their data management predictions for 2021. The experts featured here represent the top data management solution providers with experience in this niche. Data management predictions have been vetted for relevance and ability to add business value as well. These are the 17 best predictions from the dozens we received. We believe these are actionable and may impact a number of verticals, regions, and organization sizes.

Rapid evolution in how we think about the data warehouse

“Data warehouses are not going to disappear. Data warehouses will continue to be an important legacy technology that organizations will use for mission-critical business application well into the future. With the transition to the cloud, data warehouses got a fresh new look and offer some modern attractive capabilities including self-service and serverless. With the rise of the cloud, data lakes are the new kid on the block. Data lakes are becoming a commodity, legacy technology in their own right. Their rapid emergence from the innovation stage means two things going forward.

First, organizations will demand simpler, easier to manage, and more cost-effective means of extracting usable business intelligence from their data lakes, using as many data sources as possible. Second, those same organizations will want the above benefit to be delivered via tools that do not lock them into proprietary data management platforms. In short, 2021 will begin to see the rapid introduction and evolution of tools that allow users to keep their data lakes in one place and under their control while driving performance up and cost down.”

“In years past, chief data officers, and therefore the teams they direct and influence, have approached their position from a very technical and tactical mindset – operating primarily as a cost center for the enterprise as they lay a foundation for the future. However, as data becomes more intertwined with the fundamental success of the business, CDOs must refocus their efforts on strategy and transformation of how the business interacts with and benefits from data, rather than the technologies employed along the way. I predict that 2021 will be the defining year for the chief data officer, where we will see the role take shape and truly establish whether they will go the route of a cost or profit center.”

“As digital transformation initiatives accelerated significantly in 2020, massive volumes of structured, semi-structured, and unstructured data have become scattered throughout the enterprise. However, Gartner predicts that through 2022, only 20% of the organizations that invest in information governance will succeed in scaling that governance. To achieve full governance and audibility, businesses are turning to metadata to provide deeper context into where data came from, the entire series of code that ran on it, and where it went. With this continuing surge of data and increasing governance requirements, organizations are realizing that the ability to track and automate the management of metadata is the new growing challenge. In the coming year, I anticipate that with the sheer volume of metadata continuing to increase, enterprises and vendors alike will be looking for new, scalable ways to solve the metadata challenge, and increasingly lean on AI to make sense of it all.”

“Long gone are the times where organizations simply housed all their own data on-premise or even just within one cloud provider. Now organizations have data on-premise and are partnered with several cloud providers based on their specific needs. This reality has created a “rethink” of how data governance needs to be approached. Organizations must determine how their current data governance will be impacted and what needs to be adjusted, how to monitor data quality in the cloud, and how to manage data movement in and out of the cloud (and the massive expense that comes with that).”

Further evolution of data catalogs and unifying data access

“Organizations are looking to go beyond data catalogs to smarter systems that index all assets and provide deep analytics on asset content, relationships, and quality. Organizations will advance past database connections to make data connectivity to local files, cloud data/files, through APIs, from data streams, etc. all look the same to end users.”

“Distributed analytical databases and affordable scalable storage are merging into a single new thing called either a unified analytics warehouse or a data lake house depending on who you’re talking to. Data lake vendors are scrambling to add ACID capabilities, improve SQL performance, add governance, resource management, security, lineage, all the things that data warehouse vendors have been perfecting for the last three or four decades. During the ten years, while data lake software has been coalescing, analytical databases have seen their benefits and added them to their existing stacks: unlimited scale, support for widely varied data types, fast ingestion of streaming data, schema-on-read, and machine learning capabilities. Just like a lot of things used to claim to be cloudy before they really were, some vendors will claim to be a unified analytics warehouse when they’ve just jammed the two architectures together into a complicated mess, but everyone is racing to make it happen for real. I think the data warehouse vendors have an unbeatable head start because building a solid, dependable analytical database like Vertica can take ten years or more alone. The data lake vendors have only been around about ten years, and are scrambling to play catch-up.”

Data in motion will come full circle to fuel digital initiatives

“To date, data automation has mostly been about collecting “exhaust” to power BI and analytics outcomes. Going forward, the “exhaust” (and more of it than ever before) will be captured and used to feed customer initiatives in order to deliver better, more personalized outcomes and value propositions.”

AI/ML will play a bigger role n 2021 in helping make more and more tasks “invisible.”

“In our integration space, AI to date has been primarily around learning and assisting to streamline complex integration tasks (such as data mapping, for example). In 2021, machine learning will continue to improve to the point where the AI will be doing less “assisting” and more “augmenting” by actually undertaking tasks which today are normally done by a human, with AI “assistance,” via suggestions. As the AI does more and more of this, these tasks will become invisible and “just done for you,” bringing us one step closer to autonomous, self-driving integration.”

Now more than ever, automated data integration is the only way forward

“More and more, we’re seeing an enormous uptick in the volume and sources of data companies need to process. We are now measuring things in zettabytes, which is about a million petabytes and there’s no going back. Data is only going to continually increase in your organization. The only way to handle that is through automation.

In 2021, we’re going to continue to see adoption of new technologies and product integrations that help improve workflows for data analysts. With the increases in data volume, so comes the need for tools that can support more sophisticated data operations. One great example of this is dbt, the popular open-source analytics engineering tool, and I think it will play an increasingly important role in the modern data stack in 2021 and beyond.”

With the proliferation of new technology such as 5G, there will be more real-time data being generated that requires data integrity to cope with dynamism and data intelligence

“As a result, we’ll continue to see businesses double down on ensuring the quality and accuracy of their data. In the year ahead, organizations that may understand the importance of accuracy and consistency will evolve to emphasize the critical dimension of context, such as location or demographics, to set themselves apart. It will drive a greater degree of confidence needed for taking necessary and calculated risks to create new possibilities. We’re already seeing data integrity become a major initiative across industries and for the businesses that have been slower to act on it, developing a sound strategy through the effective use of data will be a priority for establishing a competitive edge in 2021.”

“Transformation is inevitable. CDOs who can build a factory of data products and bring value in digital transformation to their shifting businesses will be heroes. They will be the people who extend an olive branch to technical audiences because data products cannot be built without technical support. The CDO will have a dual role in their company, fostering a good relationship with IT and as well as mastering those new data architectures.”

“In the industry, we often talk about breaking down data silos, but we should acknowledge that some silos will always be there. In large organizations, you will always have local departments or regions that have their own tools or databases, and that will continue. If you have data sovereignty, that local office in your organization will have a silo. That’s why the best approach is to look at how you can have a better understanding of the data you have. A data intelligence platform can serve as your index and your map, showing you the silos you have and how they are connected by providing a 360-degree view of data assets.”

“Digital transformation, driven at breakneck speed due to the remote working requirements of the COVID-19 pandemic will force enterprises to accelerate data governance and compliance projects in 2021. With such a large number of remote workers, data is undoubtedly being transmitted and shared outside physical boundaries. This creates privacy and security vulnerabilities that are a boon to bad actors looking to steal personally-identifying information. Organizations must be prepared for a new normal and accommodate an endless burden on IT security teams forced to secure data without compromising remote team productivity.”

The database market will grow to $1 trillion by 2025

“For the last two decades, there’s been an iron grip on the database market with IBM, Oracle and SAP HANA leading the charge. Now we are seeing a changing of the guard, which gives customers the option of deciding what is best for their business. Forrester even points out that the public cloud infrastructure market will grow 35% in 120 billion in 2021.I predict that the database market cap will grow to $1 trillion by 2025 and over seven to 10 really strong database companies will grow significantly in the next decade.”

One single SQL query for all data workloads

“The way forward is based not only on automation, but also on how quickly and widely you can make your analytics accessible and shareable. Analytics gives you a clear direction of what your next steps should be to keep customers and employees happy, and even save lives. Managing your data is no longer a luxury, but a necessity–and determines how successful you or your company will be. If you can remove complexity or cost of managing data, you’ll be very effective. Ultimately, the winner of the space will take the complexity and cost out of data management, and workloads will be unified so you can write one single SQL query to manage and access all workloads across multiple data residencies.”

“GDPR gave rise to a “traditional” approach to governance in many organizations in 2016. Four years later, it’s clear that those efforts have largely failed. Meanwhile, an “active” approach emerged, where governance efforts are prioritized by activity & policies are put into action where data is actually used. This active approach was first attempted by innovators and early adopters, attracted by its novelty and logic—and they achieved exceptional results. Such success has been attracting increasingly conservative institutions and in 2021, we’ll see the beefy part of the bell curve adopt this increasingly vetted and popular approach.”

Expect more enterprises to declare the battle between data lakes and data warehouses over in 2021 – and focus on driving outcomes and modernizing

“Data warehouses can continue to support reporting and business intelligence, while modern cloud data lakes support all analytics, AI and ML enablement far more flexibly, scalably, and inexpensively than ever – so enterprises can go transform quickly.

Cloud migrations and related cloud data lake implementations will get demonstrably faster and easier as DIY approaches are replaced by turnkey SaaS platforms. Such solutions will slash production cloud data lake deployment times from months to minutes, while controlling costs and providing the continuous operations, security and compliance, AI and ML enablement, and self-service access required for modern analytics initiatives. That means that migrations that used to take 9-12+ months are complete in a fraction of the time.”

Images Powered by Shutterstock