Big Data Industry Predictions for 2021 - insideBIGDATA

• Read original article here

2020 has been year for the ages, with so many domestic and global challenges. But the big data industry has significant inertia moving into 2021. In order to give our valued readers a pulse on important new trends leading into next year, we here at insideBIGDATA heard from all our friends across the vendor ecosystem to get their insights, reflections and predictions for what may be coming. We were very encouraged to hear such exciting perspectives. Even if only half actually come true, Big Data in the next year is destined to be quite an exciting ride. Enjoy!

The “analytic divide” is going to get worse. Like the much-publicized “digital divide” we’re also seeing the emergence of an “analytic divide.” Many companies were driven to invest in analytics due to the pandemic, while others have been forced to cut anything they didn’t view as critical to keep the lights on – and a proper investment in analytics was, for these organizations, analytics was on the chopping block. This means that the analytic divide will further widen in 2021, and this trend will continue for many years to come. Without a doubt, winners and losers in every industry will continue to be defined by those that are leveraging analytics and those that are not. – Alan Jacobson, Chief Data and Analytics Officer, at Alteryx

Likely gone are the days of piecemeal analytics and reporting solutions that are likely fulfilling niche business use cases. This is unsustainable. Companies cannot have highly departmentalized analytics implementations that have the effect of localized problem solving and the larger business not seeing the full benefit. This current situation will change into one where analytics will be done on all data that the company has access to, with the capability of these analytics be implemented in a collaborative manner by a variety of interest groups with different skills sets (e.g., data science, lines of business leaders) and with a full-on focus towards operationalizing analytics insights in near real time. In other words, no more piecemeal and no more just science experimentation. – Sri Raghavan, Director, Data Science and Advanced Analytics Product Marketing at Teradata

Prescriptive analytics will be a key component for digital transformation success: Advanced analytics are becoming mainstreamed as businesses increasingly collect and analyze data across their organizations, with 35% of U.S. manufacturers deploying advanced analytics in the past three years. For AI to have a significant impact across the value chain, prescriptive analytics will be the catalyst to optimize performance. Prescriptive analytics will become an essential piece for scaling AI within organizations, by leveraging product and customer data to advise AI models on how to improve processes, adjust production and increase efficiency. Prescriptive analytics enables constant improvement with an AI model by continuously monitoring and adjusting based on evolving conditions. Prescriptive models can then enable decision automation, where the models can take the best course of action based on prescriptions. Going beyond predictive analytics to prescriptive analytics will ultimately enable digital transformation success for manufacturers in 2021. – George Young, Global Managing Director of Kalypso

Augmented analytics and self-service will become more widely in demand given the distributed workforce and hunger for information. In response, traditional analytics will become increasingly disrupted by AI. The increase in a distributed workforce is going to create a greater demand for augmented analytics where the individual user is guided through the process of creating queries to get immediate answers to their data questions. We are seeing a converge of analytics and AI in two areas – at the infrastructure level and at the analyst level. People are beginning to realize that they have different data pipelines that are providing data for an analytics engine and they are building a different stack for ML. Instead of two completely separate stacks, we see a convergence of these into an infrastructure that is easier to maintain while ensuring that the same data is being used to supply both engines . A second convergence will happen regarding a ‘hunger’ for information and bridging a gap to answer questions using data. Traditional analytics will start to get more disrupted by AI. Platforms (such as Tableau, Power BI, etc.) will start get displaced by bots and virtual assistants that will be conversational in nature. We see this as a push to speed through a pull for self-service. We also anticipate NLP becoming more widely used in 2021. – Scott Schlesinger, Global Data, Analytics & AI Practice Leader at Ness

The lines between IT and other departments when it comes to data and analytics in particular will continue to blur. Data and analytics have the potential to drive extremely positive and meaningful business outcomes, and when it happens, there is often also powerful collaboration across different functional areas as each one has a level of accountability for the success of the analytics approach. Areas like data governance, data literacy, open data platforms, integration and utilization of data in different parts of the enterprise will enable business users to perform tasks traditionally reserved for IT teams and the data that business units generate will feed into platforms that IT manages. This — coupled with a shortage of data scientists and analytics professionals — also means that data platforms will become more seamless and easy to deploy so that all parts of an organization will be able to leverage it. – Frances Zelazny, CMO of Signals Analytics

In the 2000s, putting Microsoft Office on your resume could make you a good candidate for a job, but a decade later it was a skill that was taken for granted. Nowadays, SQL proficiency can make you stand out, but what will happen in the years ahead?

As data literacy rises, analytics skills will become the norm for all business professionals and start to disappear from candidates’ resumes. Just as you’re unlikely to see ‘Office proficiency’ today, you’re unlikely to see ‘data proficiency’ by the end of the decade. We’ve entered a third wave of analytics, and with it the expectation that business users can interact with data without the help of an expert. Very soon, if you’re unable to marry hard data with business context to define and execute a strategy, you’re going to struggle in the workplace. The ideal candidate for businesses in 2021 and beyond will be a person who can both understand and speak data — because in a few short years, data literacy will be something employers demand and expect. Those who want to get ahead are acquiring these talents now. – ThoughtSpot CEO Sudheesh Nair

As companies shift their data infrastructure to a federated (one engine queries different sources), disaggregated (compute is separate from storage is separate from the data lake) stack, we’ll see traditional data warehousing and tightly coupled database architectures relegated to legacy workloads. But one thing will remain the same when it comes to this shift – SQL will continue to be the lingua franca for analytics. Data analysts, data engineers, data scientists and product managers along with their database admins will use SQL for analytics. – Dave Simmen, Co-founder and Chief Technology Officer (CTO), Ahana

Organizations everywhere are escalating their use of analytics systems but are challenged with the need to for event-data platforms that can perform real-time data wrangling. In 2021 organizations will demand intelligent data platforms that can consume static and streaming data from a variety of sources in any format, size or velocity; Wrangle the data (enrich and map) on-the-fly; and deliver the data to systems, devices and applications securely and in real time. – Sean Bowen, CEO of Push Technology

One single SQL query for all data workloads. The way forward is based not only on automation, but also on how quickly and widely you can make your analytics accessible and shareable. Analytics gives you a clear direction of what your next steps should be to keep customers and employees happy, and even save lives. Managing your data is no longer a luxury, but a necessity–and determines how successful you or your company will be. If you can remove complexity or cost of managing data, you’ll be very effective. Ultimately, the winner of the space will take the complexity and cost out of data management, and workloads will be unified so you can write one single SQL query to manage and access all workloads across multiple data residencies. – Raj Verma, CEO of SingleStore

AI and Analytics capabilities were provided by different platforms / teams in the past. Over the years, we are seeing the platform is converging and the AI team is more focused on the algorithmic side, while AI & Analytics platform teams merged to provide the software infrastructure for both analytics and AI use cases. – Haoyuan Li, Founder and CEO, Alluxio

As data professionals, we have a responsibility to the broader public. I think that within the next year we will see progress toward a code of ethics within the data analytics space, led by conscious companies who recognize the seriousness of potential abuses. Perhaps the US government will intervene and pass some version of its own GDPR, but I believe that technology companies will lead this charge. What Facebook has done with engagement data is not illegal, but we’ve seen that it can have deleterious effects on child development and on our personal habits. In the coming years, we will look back on the way companies used personal data in the 2010s and cringe in the way we do when we see people smoking on a plane in films from the 1960s. – Jeremy Levy, CEO ofIndicative

Emotion is a key factor affecting customer behavior and has a strong influence on brand loyalty. Therefore, it is increasingly useful for companies to find a way to measure emotions of customers during their decision-making processes. Emotional analytics focuses on studying and recognizing the full gamut of human emotions that includes mood, attitude and personality. It employs predictive models and AI/ML to analyze human movements, word choices, voice tones, and facial expressions. Emotional analytics can help companies build a more holistic customer profile, understand how to influence emotions and develop customized product and services tailored to individuals. Sentiment analysis about products and services, across geographies, social networks, and review web sites enables companies to better understand and improve their customer satisfaction level. Using emotional analytics, companies can better understand how their marketing and services influence emotion in order to provide more positively engaging customer experiences. – Paul Moxon, SVP, Data Architecture at Denodo

As businesses look toward goals to reopen and recoup sufficient revenue streams, they’ll need to leverage smart technologies to gather key insights in real-time that allow them to do so. Adopting artificial intelligence (AI) technologies can help guide companies to understand if their strategies to keep customers and employees safe are working, while continuing to foster growth. As companies recognize the unique abilities of AI to help ease corporate policy management and compliance, ensure safety and evolve customer experience, we’ll see boosted rates of AI adoption across industries. – Hillary Ashton, EVP and Chief Product Officer at Teradata

In 2021 we will see AI, machine learning and IoT define and shape our lives and behaviors, a phenomenon that will continue for many years to come. These advancements impact how we work, how we buy, how we spend, how we do every little thing in our lives. But I think the real star that companies will turn to will be the enabling technologies such as cloud and edge computing, which will continue to dominate due to their ability to process and manage all the necessary data that fuels AI, ML, and IoT, as well as enabling technologies like iPaaS, APIM and RPA. These technologies will continue to lead the digital transformation charge for businesses as they move from manual or paper-driven business to digital businesses that can finally tap the power of AI and IoT. – Manoj Choudhary, CTO at Jitterbit

Artificial Intelligence becomes less artificial in 2021: Even with a vaccine for COVID-19 on the horizon, how people work and interact has fundamentally shifted. In the new year, remote work will continue, social distancing requirements will remain, and supply chains will continue to face disruption. This new way of life demands a new way for companies to continue operations effectively across the value chain – from the product to the plant to the end user. The use of artificial intelligence (AI) will be the standard for addressing these challenges. However, without considering how humans will interact with and leverage these new autonomous systems, AI will fail. In 2021, enterprises will take a human-centered approach to AI initiatives, understanding user needs and values, then adapting AI designs and models accordingly, which will in turn, improve adoption. Enterprises must put the same focus on people and culture as the technology itself for AI to be successful. Organizational change management (OCM) teams will be critical for driving digital transformation and AI forward by bringing people along for the change journey and setting the organization up for measurable results. Proper change management is the most important – yet overlooked – aspect of any digital transformation initiative. – George Young, Global Managing Director at Kalypso

In 2021, enterprises will move away from quick wins by relying on AI systems, to focus on lasting and meaningful business value. This change will drive deeper data literacy initiatives across organizations. It will require people to learn new skills and behave in new ways. – Sundeep Reddy Mallu, Head of Analytics at Gramener 

Most consumers will continue to be skeptical of AI. With several big consumer brands in the hot seat around questionable AI ethics, most people still don’t trust AI. For many, it’s because they don’t understand it or even realize they’re using it daily. Consumers are getting so many AI-powered services for free — Facebook, Google, TikTok, etc. — that they don’t understand what they’re personally giving up in return — namely their personal data. As long as the general public continues to be naïve, they won’t be able to anticipate the dangers AI can introduce or how to protect themselves — unless the market better educates customers or implements regulations to protect them. Despite this, there’s some evidence that we’re turning the corner on AI’s trustworthiness. Eighty-one percent of business leader respondents to Pega’s upcoming survey said they’re optimistic that AI bias will be sufficiently mitigated in five years. Businesses had better hope this turns out to be true – because as more of the public wakes up to how AI impacts their lives, and in some cases plays favorites, they will continue to ask harder questions that further erodes trust in AI, forcing businesses to have to answer to them. – Vince Jeffs, Senior Director – Product Strategy, Marketing AI and Decisioning, Pega

AI powered digital workers will help businesses stay strategic in the long-term. Few disagree with the notion that AI and automation are essential to companies’ survival going forward. However, research has indicated that most companies have not fully realized the benefit of their AI and automation investments. By linking powerful AI capabilities to business processes through the digital workforce, we’ll increasingly see organizations implement AI driven automation at scale. AI infused automation will increasingly be linked to core strategic initiatives such as improved customer focus, revenue growth, capital allocation, supply chain management, risk management, cost and operational efficiency and more. AI powered digital workers will be leveraged as primary tools for executing on corporate strategy and managing enterprise scale risks. Rapid and effective adoption of automation will increasingly be seen as an essential component to remaining competitive in markets. – Eric Tyree, Head of AI and Research at Blue Prism

AI experimentation will become more strategic. Experimentation takes place throughout the entire model development process – usually every important decision or assumption comes with at least some experiment or previous research to justify those decisions. Experimentation can take many shapes, from building full-fledged predictive ML models to doing statistical tests or charting data. Trying all combinations of every possible hyperparameter, feature handling, etc., quickly becomes untraceable. Therefore, we’ll begin to see organizations define a time and/or computation budget for experiments as well as an acceptability threshold for usefulness of the model. – Florian Douetteau, CEO and co-founder of Dataiku

In 2021, we will finally see AI go mainstream. As a result of COVID-19, businesses were forced to digitally transform in order to survive in the new normal. According to our research, digital acceleration shows no sign of stopping in the new year, with 86% of companies currently reaping the benefits of better customer experience through AI, likely to continue. The pandemic has also changed business priorities for AI investment. For instance, we’ve seen companies shift from simpler tasks like automation to focus on workforce planning and simulation modeling. As organizations continue to see benefits from their digital investments in complex processes, AI will only become more widespread and widely used over the next year. – Anand Rao, Global Artificial Intelligence Lead at PwC

Convergence of AI & BI will boost data insights. AI has been part of every corporate discussion over the past 5 years. And yet, challenges persist in democratizing advanced AI insights across large sections of employees. As new AI-powered BI products emerge, silos will be broken and every user will be able to leverage data analytics and find insights easily. Simple interfaces, personalized insights, and engaging data experiences will become the hallmarks of data analytics in 2021 and beyond. – Dhiren Patel, MachEye’s Chief Product Officer & Head of Customer Success

Racial bias in many AI-driven facial recognition algorithms has been a big topic of conversation over the past year and came to a head due to the social unrest of 2020. Research has found widespread evidence that racial minorities were far more likely than whites to be misidentified. In 2021, we will see the correction of AI bias become a major topic for any company that leverages AI or facial recognition technology. By using government-issued documents, you can quickly and easily prove ID ownership by analyzing the face on the document and comparing it to the face trying to access your system. 2021 will be the year that AI bias comes to light and companies will begin implementing radical change to eliminate racial bias in its software — some of which can be done by putting a deliberate focus on fairness and training of the company’s ML system to reduce racial facial recognition errors. – Mohan Mahadevan, VP of Research, Onfido

2021 will be the year that teams go from casually dating AI to being in a committed relationship. AI isn’t just for R&D projects anymore. It’s time to commit to adapting these solutions instead of just flirting with them. We have to automate now. – David Karandish, Founder and CEO of Capacity 

With the confluence of computational power, internet-scale data and modern machine learning algorithms, we have broken remarkable new ground with AI over the past few years. In the coming years, we will enter an expansionary era, where a long tail of commercial use-cases will be prototyped, packaged and productionized – either to enhance existing products and services or to create entirely new ones. – Dave Costenaro, Chief Data Officer at Capacity 

AI Success Moves From General Purpose to Niche Focuses. While AI investment continues to grow in the enterprise, businesses are reevaluating their tech stacks to accommodate niche AI, rather than “general purpose” black boxes that claim to do everything. Niche, perfected use-cases that solve specific problems are going to take budget priority, rather than automation which promises to do everything. – Viral Bajaria, CTO at 6sense

Rise of Artificial Narrow Intelligence: Not long ago, AI was what we now know as artificial general intelligence, like self driving cars or image recognition. However, today there is a new category of artificial narrow intelligence which is trying to replicate a human decision making process. From a supply chain perspective, this new AI can help to inform better decision making around every aspect of a supply chain, from “How do I fill a truck?” or “How do I get products on time?” In 20201, I’m envisioning an increase in these narrow solutions to replace tactical and smaller scale decisions. – Andy Fox, Director of Global Impact with LLamasoft

At the fringes, we will begin to see “Counter-AI” start to materialize. As governments try to track people and businesses try to manipulate them or gain deep insights into behavior, I predict a backlash of methods to foil tracking and customer 360’s. Not unlike the work various groups have done on anti-facial recognition tools, we will begin to see high and low-tech methods for boggling the AI’s used to monitor and understand us. – Head of Architecture for Atos North America’s AI Lab in partnership with Google Cloud, Jonas Bull

As more agencies begin to adopt these AI- and ML-based solutions, there’s an onus on law enforcement to abide by ethical policies and to remove bias in such tools. As such, departments will begin to establish their own policies and work with governing bodies on responsible and ethical AI usage, including proper training for the relevant teams and business functions, as well as creating an environment with an ethos of data-driven and responsible decision-making. Going a step further, law enforcement organizations will continue to ensure AI systems are vetted to be bias-free and corrected as needed. And they will open a line of communication with the public to promote transparency regarding the use of these tools. – Heather Mahalik, Senior Director of Digital Intelligence, Cellebrite

We’ll see more data-driven companies leverage open source for analytics and AI in 2021. Open source analytics technologies like Presto and Apache Spark power AI platforms and are much more flexible and cost effective than their traditional enterprise data warehouse counterparts that rely on consolidating data in one place–a time-consuming and costly endeavor that usually requires vendor lock-in. Next year will see a rise in usage of analytic engines like Presto for AI applications because of its open nature – open source license, open format, open interfaces, and open cloud. – Dipti Borkar, Co-founder and Chief Product Officer (CPO), Ahana

The industry will shift away from generic horizontal AI platforms, such as IBM Watson and Amazon Lex, towards domain specific AI powered products and managed service models. Generic platforms are not solutions, They start cold, without any training data or data model structure — building this, then optimizing it in production is an expert and resource intensive task that is beyond most companies capability. The move from the early innovator market into mass market adoption will be driven in 2021 by the adoption of domain specific AI powered products that are pre-trained for a specific industry and are proven to work. – Jake Tyler, co-founder & CEO, Finn AI

In 2021, AI won’t be mapped on the human spectrum of competence. We can have algorithms that crush any human at chess but are unable to make a cup of tea and computer programs that can perform mathematics millions of times faster than humans but, if asked who might win the next World Cup, they wouldn’t even understand the question. Their capabilities are not universal. We’ve reached a point with AI where we simultaneously overestimate and underestimate the power of algorithms. When we overestimate them, we see human judgment relegated to an afterthought – a dangerous place to be. The use of a “mutant algorithm” in grading A-level results is the scandal du jour in the UK, despite the algorithm producing many results that simply violate common sense. When we underestimate algorithms, we see entire industries crumble because they didn’t see change on the horizon. How can the traditional taxi business compete when Uber’s algorithm can get you a ride in less than 3 minutes? In 2021, expect engineers to avoid AI and algorithmic blunders by not trying to map algorithms onto the human spectrum of competence. Using AI technologies – such as any-context speech recognition – to enhance what humans can do and finding the right balance between AI automation and human knowledge for real world use cases – such as customer experience and web conferencing – will begin to shape the effective use of AI for the future. – Ian Firth, VP at Speechmatics

Responsible AI / ML will become the hottest topic in the cloud ML industry. Given society’s increased emphasis on combatting unfairness and bias and the overall interest in better interpretability and explainability of machine learning models, cloud providers will invest and enhance their ML offerings to offer a full suite of responsible ML / AI capabilities that will aim to satisfy and reassure regulators, modelers, management and the market on the fair use of ML. Meanwhile, AI / ML will continue to see explosive growth and usage across the whole industry, with significant enhancements in ease-of-use and UX combining within a responsible AI / ML framework to drive the next growth spurt of this sector. – Yiannis Antoniou, analyst, Gigaom

AIOps for networking will become mainstream: Next year, AIOps will go from theory to practice for many organizations. With the increase of remote workers and the home becoming the new micro branch, AI will become table stakes for delivering a great client to cloud user experience while controlling IT support costs for remote employees. IT teams will need to embrace AIOps to scale and automate their operations. AIOps cloud SaaS will turn the customer support paradigm upside down. Instead of users submitting tickets to IT, AI will proactively identify users with connectivity or experience issues and will either resolve (the self-driving network) or will open a ticket with suggested remediation actions for IT. – Bob Friday, CTO of Mist Systems, a Juniper Networks company

Artificial intelligence and machine learning will play a much more integral role in supply chain strategy than in previous years. The need for more real-time insights throughout the supply chain will continue to grow in 2021, especially as supply chain organizations re-evaluate their operations as a result of sudden changes in buying behaviors during the COVID-19 pandemic. To address this need, supply chain organizations will need to look to artificial intelligence (AI) and machine learning (ML) enabled technology to upgrade from current, descriptive and prescriptive analytics, and leverage predictive analytics — which provide recommended actions beforean incident occurs based on previous actions. Oftentimes, companies experience a mess of silos and fragmentation due to being acquired by large companies that have different systems. In 2021, supply chain stakeholders will look to deploy digital twins across all modules as an extra layer of visibility and to ensure synchronization between a company’s existing systems and new technology, such as sensors and nano sensors, which are coming to market in increasingly larger volumes. – Mahesh Veerina, CEO of Cloudleaf

Bias in AI causes harm at great scale – from impacting the recruitment process by reinforcing gender stereotypes to racial discrimination in credit scoring and lending. Organizations know that hiring a diverse workforce can provide a level of truth for AI models, and they know that training data needs to be constantly monitored for bias, as it impacts the quality and accuracy of algorithms. They also know that there is no current benchmark for ethics-based measurements to truly mitigate bias in AI, and that there needs to be. In 2021, we’ll see organizations moving past just acknowledging and “worrying” about bias in AI and start to make more significant moves to solve for it – because it will be required. Specific teams and/or initiatives will be formed to combat all the concerns that fall under the umbrella of responsible AI, including everything from inherent bias in data to treating data trainers fairly. Establishing responsible AI initiatives will not only become a board-level mandate for some, but the partners and customers of companies leading AI efforts will demand it. – Appen CTO Wilson Pang   

AIOps Will Heat Up to Enhance the Customer Experience and deliver on Application Assurance and Optimization. With a year of unpredictability behind us, enterprises will have to expect the unexpected when it comes to making technology stacks infallible and proactive. We’ll see demand for AIOps continue to grow, as it can address and anticipate these unexpected scenarios using AI, ML, and predictive analytics. The increasing complexity of digital enterprise applications spanning hybrid on-premise and cloud infrastructures coupled with the adoption of modern application architectures such as containerization will result in an unprecedented increase in both the volume and complexity of data.  While data overload from modern digital environments can delay repair and overwhelm IT Ops teams, noisy datasets will be a barrier of the past as smarter strategies and centralized AIOps systems help organizations improve the customer experience, deliver on modern application assurance and optimization, tie it to intelligent automation, and thrive as autonomous digital enterprises. In fact, conventional IT Operations approaches may no longer be feasible – making the adoption of AIOps inevitable to be able to scale resources and effectively manage modern environments. – Ali Siddiqui, Chief Product Officer, BMC Software

The stark reality is that 2021 will be the year when those actually doing AI will start achieving value at scale, while those spending months training brittle models and failing to catch up will be at an increasing, exponential, disadvantage. Last mile challenges won’t get any easier – but a fundamental shift in thinking and approach will be critical to overcoming complexity obstacles. – Dr. Josh Sullivan, Head of Modzy 

Elegant risk assessment: As the AIOps space continues to mature, we see an opportunity for vendors to refine their risk assessment capabilities to enable customers to fix issues with near-certainty, without breaking anything else in the system. In 2021, one area where we will see increased focus from both vendors and more adoption among users will be around enabling more elegant  dependency mapping so engineers can accurately assess risk as a part of the remediation process or build-deploy cycle for software changes, to ensure that a change in one part of an environment won’t break the system elsewhere. – Michael Olson, Director, Product Marketing at New Relic 

In 2021, AI Won’t Be Mapped on the Human Spectrum of Competence: We can have algorithms that crush any human at chess but are unable to make a cup of tea and computer programs that can perform mathematics millions of times faster than humans but, if asked who might win the next World Cup, they wouldn’t even understand the question. Their capabilities are not universal. We’ve reached a point with AI where we simultaneously overestimate and underestimate the power of algorithms. When we overestimate them, we see human judgment relegated to an afterthought – a dangerous place to be. The use of a “mutant algorithm” in grading A-level results is the scandal du jour in the UK, despite the algorithm producing many results that simply violate common sense. When we underestimate algorithms, we see entire industries crumble because they didn’t see change on the horizon. How can the traditional taxi business compete when Uber’s algorithm can get you a ride in less than 3 minutes? In 2021, expect engineers to avoid AI and algorithmic blunders by not trying to map algorithms onto the human spectrum of competence. Using AI technologies – such as any-context speech recognition – to enhance what humans can do and finding the right balance between AI automation and human knowledge for real world use cases – such as customer experience and web conferencing – will begin to shape the effective use of AI for the future. – Ian Firth, VP at Speechmatics

In 2021, open and free data collection will fuel future innovations. A recent survey from Frost & Sullivan found that 54% of IT decision-makers expressed a need for large-scale data collection to keep pace with their businesses’ growth and online competition. However, for businesses to utilize online data effectively, it first needs to be accessible – not blocked. Today, businesses often prohibit public data collection attempts despite collecting it themselves. This situation is caused by two major factors: the continuous need to block malicious or fraudulent online activity as part of security precautions, and the notion that this public data contributes to a company’s competitive edge. I believe that during 2021 and onwards companies will realize that public data collection is part of the general and necessary ongoing business conduct. They will also realize that data isn’t everything when it comes to a business’s competitive edge. Areas such as inventory, prices, product quality, and service quality, etc., play a big role as well.  Once that realization settles in, blocking data will serve only to protect against abusive online activities. To secure ethical data collection, I hope we all promote an open exchange of information in central data hubs. Sites will continue to block abusers; this will not change. However, they may permit ethical data collectors. Ultimately, the future of online data collection is up to those who control it. At the rapid rate that data is being produced, future data collection efforts will need to evolve and grow. Companies will need automated data collection to keep up with their competitors and be able to gather data at a faster rate.  After all, the speed at which companies can collect fresh data will determine their relevancy and success. – Ron Kol, CTO at Luminati Networks

Data will become truly operational on an enterprise scale: The amount of data businesses have is growing exponentially – there are more sources, types and amounts than ever before, plus increasing amounts of data are being delivered in near-real time. But to truly understand, access and take action on data, enterprises will need to change how they consume it — starting by cutting out the middleman. By finding ways to automate the data cataloguing and profiling processes, employees – including those with less of a technical background – will be able to get the data they need to effectively and efficiently make good business decisions. – Eric Raab, SVP, Engineering and Product, Information Builders

It’s essential to capture and synthesize “alternative” data: How early could we have detected COVID-19? Studies of “alternative” data – in this case, traffic data outside hospitals in Wuhan and keyword searches by Internet users in that area – indicate that the virus may have been circulating in late 2019. The investment community has been a pioneer in using alternative data, including audio, aerial photos, water quality, and sentiment.10 This is the front line for data-driven innovation, and getting an edge here can result in huge gains. But in the wake of 2020, alternative data will become mainstream, with the goal of spotting anomalies much earlier. From that, we can get derivative data, which comes from combinations, associations and syntheses with data from systems of record. As IDC says: “As more data gets captured and becomes available from external sources, the ability to use more of it becomes a differentiating factor. That includes taking lessons from industries other than your own.” 11 This trend, similar to what Gartner calls “X analytics,” 12 isn’t new but is finally becoming an important foundation of modern data and analytics, thanks to cheaper processing and more mature AI techniques – including knowledge graphs, data fabrics, natural language processing (NLP), explainable AI and analytics on all types of content. This trend is completely dependent on ML and AI, as the human eye can’t catch it all. – Dan Sommer, Senior Director, Global Market Intelligence Lead at Qlik

In the industry we often talk about breaking down data silos, but we should acknowledge that some silos will always be there. In large organizations you will always have local departments or regions that have their own tools or databases, and that will continue. If you have data sovereignty, that local office in your organization will have a silo. That’s why the best approach is to look at how you can have a better understanding of the data you have. A data intelligence platform can serve as your index and your map, showing you the silos you have and how they are connected by providing a 360-degree view of data assets. – Stijn “Stan” Christiaens, co-founder and CTO of Collibra

OpenTelemetry will create data overload. In 2021, the use of OpenTelemetry will become the new industry norm. Yes, it will make data collection easier by creating consistency across sources — but it will also create a data firehose for companies, making it even harder to find the small portion of data containing actionable insights. The constant stream of data will overwhelm companies if they don’t have a system in place to quickly find the 5% that is truly actionable. Because of this, IT teams will shift their focus from acquiring data to building a framework to take action from data. As teams do so, it will be imperative to implement tools that can immediately start surfacing actionable data in the time it takes to make a cappuccino. – Phil Tee, CEO of Moogsoft

Proliferation of low-code/no-code ML. The increase of low-code and no-code ML systems, designed to make AI more accessible to companies, will help improve adoption of AI. However, eventually companies will reach a ceiling and outgrow the one-size-fits-all approach, seeking more advanced use cases for AI that require deeper expertise. Ultimately, the need for customization will increase the need for qualified data scientists, rather than low-code systems replacing them. We aren’t going to automate away the need for data scientists any time soon. — Kevin Goldsmith, CTO, Anaconda

Business Intelligence is shifting to a new paradigm of advanced data analytics with the integration of Natural Language, Natural Search, AI/ML, Augmented Analytics, Automated Data Preparation, and Automated Data Catalogs. This will transform business decision-making processes with higher-quality real-time insights. – Ramesh Panuganty, CEO of BI company MachEye

BI and AI will deepen their liaison. Whether scoring BI data sets against ML models and visualizing the predictions, or leveraging natural language processing for generating visualizations, insights, and summaries, AI and BI will increase their synergies. And as conventional BI capabilities continue to commoditize, vendors will need BI+AI as a new front in the innovation wars. – Andrew Brust, analyst, Gigaom

Employee to Enterprise – Conversational AI adoption will be natural and often the first contact. Conversational AI is normalized and here to stay. Interfaces that guide consumers through the online marketplace, employees through training courses and users through search engines and websites saw great returns on investment when outfitted with advanced Conversational AI technology. – Shiva Ramani, CEO of iOPEX

AI will not displace human beings any time soon. When you look at the use of AI in consumer-facing operations today, it’s mainly used in AI-supported chatbots and customer personalization features. If we look at how consumers have taken advantage of AI-supported features during the pandemic, we can see that they’re actually using them to resolve issues faster through human agents. Companies like Bank of America, which has a consumer-facing AI-powered chatbot named Erica, saw consumers using Erica to find the best course of engaging customer support teams. Rather than asking Erica questions to fix any issues directly, customers simply asked Erica how they should go about reaching out to the customer service team to rapidly resolve their problem with the appropriate human agent. – James Isaacs, President and CEO of Cyara

Today, we interact with bots more than ever before, whether it’s customer service chatbots or the AI on our devices, like Siri and Alexa. These bots are used for real-time decision making to automate processes that were previously done by humans. For example, bots have automated the retail return processes for companies like Amazon. However, it becomes more complicated for enterprises to manage the identities of automated bots, especially when they are interacting with other bots at machine speed. The identities of bots must be managed and protected by the enterprise, similar to employee and customer identity, so that data isn’t compromised. This is important for CIOs and security leaders to keep in mind, because using bots for automation purposes will open new attack vectors if those bots’ APIs are hacked. – Jasen Meece, CEO of Cloudentity

NLP (natural language processing) changes the conversation on data analysis: Just as we are using Google Home and Alexa in our everyday lives, conversational analytics through NLP will be the golden ticket for enterprises in extracting valuable big data insights from their business operations. This includes unearthing trends that may have gone unnoticed and allowing experts from within the enterprise to engage with data in a meaningful way. – Sam Mahalingam, CTO, Altair

Conversational AI, first and foremost, needs an ubiquitous messaging channel to converse on. The rise of business messaging on IP-based channels such as Whatsapp, GIP and others is driving a resurgence in the use of Conversational AI. Companies across industries such as banking, e-commerce, retail, travel etc are now enabling conversational AI for virtually every customer touchpoint including marketing, sales and support. Powered by recent advances in natural language processing (NLP), conversational AI is poised to transform how consumers interact with businesses. – Beerud Sheth, CEO ofGupshup

I think we will begin to see a more thoughtful, balanced approach to multi- and hybrid cloud adoption, particularly for hybrid cloud. We are getting past the public versus private cloud conversations, and businesses are accepting the reality that cloud is not an “either, or” decision. Historically, we have seen “public cloud” being associated with cutting-edge innovation and “private cloud” being associated with slow, legacy businesses that are resistant to change. This sentiment is changing, as businesses are beginning to better understand the value they can get from a hybrid cloud architecture that enables them to deploy agile, modern applications on the platform that best balances their specific cost, performance, security, compliance and governance needs. With this comes an increase in hybrid enabling technologies such as containers and hybrid integration platforms. Another consideration is tethered compute, which is a hyperscale cloud provider solution running in your own data center. Examples are AWS Outposts, Google Anthos, and Microsoft Azure Stack. Although these have been too slow to adopt to date, we could start to see the beginning of growth here as customers see the value of private/public cloud, coupled with the consistency of hyperscale cloud service consumption. – Kim King, Director of Product Marketing – Cloud Management at Snow Software

COVID-19 Accelerates Cloud Spending: With the increase of remote working due to the COVID-19 pandemic, companies are investing a larger portion of IT budgets on cloud-based technologies, moving away from paper-based processes. Enterprises’ average cloud spending is up 59% from 2018 to $73.8M in 2020. That trend will continue into 2021 as companies are forced to adopt strategies to work remotely and recognize the benefits of maintaining those modes of operating even as they begin to transition employees back to physical locations. A prime example will be contracting where COVID drove digital transformation of the contract request, approval, execution, and post-award management systems and has laid the groundwork for even more advancements in contract lifecycle management. – Harshad Oak, General Manager, Customer Adoption & Value, at Icertis

The solutions that companies use to store their data continue to rapidly evolve in the next year. We are seeing increased migrations into open source relational database solutions, non-relational database solutions, PaaS-based database solutions, and a combination thereof. The primary focus of these initiatives can be grouping under the heading of reducing operating costs, whether they are being undertaken to reduce hefty support contracts from vendors like Oracle and Microsoft (both the open source and non-relational database migrations fall into this category), reduce headcount expense (migrations to PaaS services falls into this category), or gaining performance efficiencies by migrating to a more purpose-built database solution. Data migration is happening right now and at a large scale, so there are many considerations that need to be made when transitioning to these new database solutions, including the capabilities of the future state solution versus the current state, the impact to licensing and support contracts, and a method to ensure that the correct solutions are deployed. While PaaS solutions provide some great benefits, DBAs are still required to monitor and manage those systems and work with application teams to drive efficiencies in performance, availability, and security. – Marc Caruso, Chief Architect, Syntax

360. That’s the number of database systems out in the wild. And while choice is good and finding the right tool for the job is smart, it also adds major complexity. As companies move to modernize in the cloud, they will seek simplification, which will lead to massive consolidation in the database market. Database vendors that offer multi-functional capabilities will win, rather than a multitude of niche databases that need to be stitched together and require different ways of accessing data. – Franz Aman, CMO of relational database company MariaDB

The solutions that companies use to store their data continue to rapidly evolve in the next year. We are seeing increased migrations into open source relational database solutions, non-relational database solutions, PaaS-based database solutions, and a combination thereof. The primary focus of these initiatives can be grouping under the heading of reducing operating costs, whether they are being undertaken to reduce hefty support contracts from vendors like Oracle and Microsoft (both the open source and non-relational database migrations fall into this category), reduce headcount expense (migrations to PaaS services falls into this category), or gaining performance efficiencies by migrating to a more purpose-built database solution. Data migration is happening right now and at a large scale, so there are many considerations that need to be made when transitioning to these new database solutions, including the capabilities of the future state solution versus the current state, the impact to licensing and support contracts, and a method to ensure that the correct solutions are deployed. While PaaS solutions provide some great benefits, DBAs are still required to monitor and manage those systems and work with application teams to drive efficiencies in performance, availability, and security. – Marc Caruso, Chief Architect, Syntax

The database market will grow to $1 trillion by 2025. For the last two decades, there’s been an iron grip on the database market with IBM, Oracle and SAP HANA leading the charge. Now we are seeing a changing of the guard, which gives customers the option of deciding what is best for their business. Forrester even points out that the public cloud infrastructure market will grow 35% in 120 billion in 2021. I predict that the database market cap will grow to $1 trillion by 2025 and over seven to 10 really strong database companies will grow significantly in the next decade. – Raj Verma, CEO of SingleStore

The Data Lake Can Do What Data Warehouses Do and Much More

While the separation of compute and data provides advantages for data lakes over data warehouses, data warehouses have historically had other advantages over data lakes. But that’s now changing with the latest open source innovations in the data tier. For example, Apache Iceberg is a new table format that provides key data warehouse functionality in the data lake such as transactional consistency, rollbacks and time travel while introducing new capabilities that enable multiple applications to work together on the same data in a transactionally consistent manner. Another new open source project, Project Nessie, builds on the capabilities of Iceberg as well as Delta Lake by providing Git-like semantics for data lakes. Nessie also makes loosely-coupled transactions a reality, enabling a single transaction spanning operations from multiple users and engines including Spark, Dremio, Kafka and Hive. – Tomer Shiran, co-founder of Dremio

Companies will reinvest in the data engineer and data pipelines. One impact of 2020 was that a lot of companies shifted to a survival-first approach, which resulted in a “grab-and-go” mentality to their data integration. As businesses’ bottom lines are stabilizing and we’re seeing more predictability at the macroeconomic level, our prediction is that 2021 is the year of the data engineer, and that companies are going to get back to a “built to last” approach for data pipelines. “Built to last” for the water in your pipes at home means that water is always on, clean and at the right temperature. “Built to last” for data means that you build smart data pipelines to ensure timeliness and confidence in your data analytics. – StreamSets CEO Girish Pancha

IT will infuse access governance with intelligence to protect workforce cybersecurity in 2021. Accelerating changes in enterprise technologies, cyberthreats and the user landscape are increasing pressure on traditional identity governance and administration (IGA) solutions and, in turn, on security and compliance teams. On top of growing compliance risks, enterprise IT environments become more complex every year, increasing the number of applications and systems to which companies provide user access. These challenges are driving organizations to seek out AI-driven solutions that simplify and automate the access request, access approval, certification and role modeling processes. In 2021, we will see AI increasingly employed to enable an autonomous identity approach. AI-infused authentication and authorization solutions will be layered on top of, or integrated with, existing IGA solutions, providing contextual, enterprise-wide visibility by collecting and analyzing all identity data, and enabling insight into different risk levels of user access at scale. The use of AI will allow systems to identify and alert security and compliance teams about high-risk access or policy violations. Over time we will see these AI systems produce explainable results while increasing automation of some of the most difficult cybersecurity challenges inside the enterprise. – Eve Maler, CTO at ForgeRock

We have seen the global implementation of AI governance frameworks take off in 2020 where enterprises are asking for details on the outcome of AI applications. Ensuring an appropriate level of explainability of AI applications is key as well as using good quality data, ensuring auditability, being ethical, fair and transparent, complying with data protection requirements, and implementing effective cybersecurity measures. Implementation of AI governance frameworks is seen more in financial and banking currently, but in 2021 we’ll see this become more widespread. Other verticals such as healthcare, e-commerce and mobility services will begin to use it as a competitive differentiator. For instance, healthcare providers are beginning to be more transparent with how data is used, and how they are ethical and fair in protecting that data. If businesses want to stay ahead of the curve, they should start developing ethical AI frameworks now in order to position themselves as a leader in this global movement. – Mohan Mahadevan, VP of Research, Onfido

AI Will Gain Momentum in Cloud Security and Governance. In 2021, AI will go far beyond simply detecting anomalies and thereby flagging potential threats to security teams. Cloud governance is an increasingly complex task and is quickly reaching a point where it’s impossible for humans to manage alone. AI will increasingly be relied on in the coming year to maintain cloud hygiene by streamlining workflows, managing changes and archiving. Once proper cloud hygiene is established and maintained with AI, it will also be used as a strategic predictive knowledge tool. By predicting and addressing threats and vulnerabilities, AI will help enterprises create the best possible outcome for their cloud environments. Leveraging AI as a strategic asset will empower CIOs to make informed decisions about their cloud environments, such as evaluating costs and compliance risks. – Keith Neilson, Technical Evangelist for CloudSphere

As we look to 2021, we will see the conversation of ethical AI and data governance be applied to multiple different areas, such as contact tracing (fighting COVID-19), connected vehicles and smart devices (who owns the data?), and personal cyber profiles (increased cyber footprint leading to privacy questions). – Cindy Maike, VP of Industry Solutions, Cloudera

Data governance for a multi-environment reality. Long gone are the times where organizations simply housed all their own data on-premise or even just within one cloud provider. Now organizations have data on-premise and are partnered with several cloud providers based on their specific needs. This reality has created a “rethink” of how data governance needs to be approached. Organizations must determine how their current data governance will be impacted and what needs to be adjusted, how to monitor data quality in the cloud, and how to manage data movement in and out of the cloud (and the massive expense that comes with that). – Todd Wright, Head of Data Management and Data Privacy Solutions at SAS

AI Will Gain Momentum in Cloud Security and Governance. In 2021, AI will go far beyond simply detecting anomalies and thereby flagging potential threats to security teams. Cloud governance is an increasingly complex task and is quickly reaching a point where it’s impossible for humans to manage alone. AI will increasingly be relied on in the coming year to maintain cloud hygiene by streamlining workflows, managing changes and archiving. Once proper cloud hygiene is established and maintained with AI, it will also be used as a strategic predictive knowledge tool. By predicting and addressing threats and vulnerabilities, AI will help enterprises create the best possible outcome for their cloud environments. Leveraging AI as a strategic asset will empower CIOs to make informed decisions about their cloud environments, such as evaluating costs and compliance risks. – Keith Neilson, Technical Evangelist for CloudSphere

2020 was brutal for some firms, rewarding for others, and challenging for all. As we enter 2021, laggards have an existential imperative to reinvent themselves digitally, leading firms struggle to keep pace with demands. All of these enterprises need to capitalize on 100% data integration with predictable costs, reliable performance and real-time visibility. – Bonnie Holub, Practice Lead, Data Science, Americas at Teradata

Data democratization will become the new norm. It’s the job of the CDO to ensure expansion of growth across the entire business. This can be achieved by providing structured data that people can actually use. A successful CDO should democratize data so that it’s accessible and understandable by people. A good CTO will complement the CDO by creating the necessary tooling to find the required data. This means giving users a set of visualization tools and reporting tools that allow them to get after the data to run insights. As we move into 2021, we’ll continue to see further and tighter collaboration between these two roles, driven by necessity. If you have tools with bad data, you’re exacerbating the data challenge. If you have limited tools, only a small subset can do anything with the data. – Derek Knudsen, Chief Technology Officer at Alteryx

Citizen analysts will increasingly up-skill to become data scientists. The growing complexity of most industries and companies also means that once we see self-reliance in terms of developing IT processes or using analytics, there will quickly be a huge push to expand that skill-set further. With the market erratically changing from month to month there will be a much greater emphasis placed on data science than ever before. This, in turn, will drive more citizen analysts to up-skill to become data scientists. – Sharmila Mulligan, Chief Strategy and Marketing Officer at Alteryx

Python data visualization libraries will sync. We’re finally starting to see Python data visualization libraries work together, and this work will continue in 2021. Python has had some really great visualization libraries for years, but there has been a lot of variety and confusion that make it difficult for users to choose appropriate tools. Developers at many different organizations have been working to integrate Anaconda-developed capabilities like Datashader’s server-side big data rendering and HoloViews’ linked brushing into a wide variety of plotting libraries, making more power available to a wider user base and reducing duplication of efforts. Ongoing work will further aid this synchronization in 2021 and beyond. — James A. Bednar, Sr. Manager, Technical Consulting, Anaconda

Business skills will become more critical than ever for data scientists. Data scientists will need to speak the language of business in order to translate data insight and predictive modeling into actionable insight for business impact. Technology owners will also have to simplify access to the technology, so that technical and business owners can work together. The emphasis for data scientists will be not just on how quickly they can build things, but on how well they can collaborate with the rest of the business. – Florian Douetteau, CEO and co-founder of Dataiku

Shared data, visualizations and storytelling are consumed by the masses: In a virtual world, self-service needs to evolve. When there are no instruction manuals and no one there to hold a user’s hand, a fast, intuitive ramp-up becomes a hygiene factor for adoption, and compelling user interfaces will no longer be a nice-to-have. But we’ve also seen that users often don’t want to self-serve; they increasingly expect insights to come to them. As a result, we’ll see more micro-insights and stories for the augmented consumer. In addition, data is too often overlooked. Empowering users to access data, insights and business logic earlier and more intuitively will enable the move from visualization self-service to data self-sufficiency. AI will play a major role here, surfacing micro-insights and helping us move from scripted and people-oriented processes to more automated, low-code and no code data preparation and analytics. If more people can be self-sufficient with data earlier in the value chain, anomalies can be detected earlier and problems solved sooner. – Dan Sommer, Senior Director, Global Market Intelligence Lead at Qlik

Historically companies put a lot of value on people who were “Data Scientists”. Going forward, there will be a need to hire people that are experts in data collection. For AI models to work, vast amounts of data is required, and moreover, critical data still resides in silos in many organizations; hence, individuals with skills in data collection will be high in demand. – Clara Angotti, President of Next Pathway

Data scientists will play a critical role in the development of a COVID-19 vaccine. From development of a vaccine to analysis of trials and deployment, data will be the key to knowing if we have found a preventative solution. Data scientists will be as important as traditionally trained scientists in producing the first viable vaccine. To accelerate the development of vaccines, people must be able to manage, make decisions and trust that data. Knowing that speed is critical, data agility is required and new automated systems will enable new innovations, ultimately leading to a vaccine. Accelerating the delivery of the vaccine will require a great deal of agility and automation in managing data. – Infoworks CEO Buno Pati.

While data  continues to rule the world, organizations are still finding themselves struggling to leverage that data for a true competitive advantage. The Citizen Data Science Movement has emerged to widely promote the ability to manipulate and interpret data. But is there is a better way? Wouldn’t it be smarter (and easier) to simply bring business meaning to the data and repair the data rather than fix the people given that raw uninterpreted data located somewhere in a system isn’t very helpful. – Kendall Clark, founder & CEO of Enterprise Knowledge Graph Platform developer, Stardog

The adoption of Deep Learning based enterprise solutions in startups and enterprises will see a gradual uptick. The key hindrance will continue to be the costs of procuring GPU instances and high-cost human resources. – Sundeep Reddy Mallu, Head of Analytics at Gramener 

As we all witnessed in recent years, research and development in Natural Language Processing has progressed rapidly by breakthroughs in Transformer language models such as BERT, GPT-3 etc. While they are achieving state-of-the-art performance, they require large datasets and large amounts of computational resources for training and inference with a significant carbon footprint. We will see more efforts and research coming out with new model architectures and training techniques to address the concerns of carbon emissions, very long training times, with space and compute effective models to make these breakthroughs more accessible; recent models like Performers with Fast Attention will serve as catalysts to move in this direction. – Kavan Shukla, Data Scientist, Finn AI 

Hardware and software converge with the rise of AI-specific hardware. As Apple’s announcement of the M1 chip showed, purpose-built hardware is becoming more mainstream, meaning that people will begin to think more about the actual hardware that they are working on than they did previously—including data scientists. The rise in ML-specific hardware will likely lead to performance improvements, but also provides another variable in model deployment. It’ll be particularly impactful in cloud and mobile environments. This will further break down the wall that has traditionally existed between hardware and software, with AI use cases leading the way. — Kevin Goldsmith, CTO, Anaconda

Since 2012 AI compute power has grown at 5X the rate of Moore’s Law, doubling approximately every 3.5 months. Given the growing number of applications built on top of AI engines impacting our everyday lives – some even critical to humanity as a whole (e.g. modeling and solving for climate change), finding a solution to this performance scaling mismatch is high on every serious fabless and chip manufacturing company’s priority list. The need for shifts in how Moore’s Law is perceived will become more apparent in 2021. The latest trend has been to talk about writing more efficient software to yield year-over-year perfor