Streaming Data Success Relies on Managing Data in Motion and At Rest

I recently noted that as demand for real-time interactive applications becomes more pervasive, the use of streaming data is becoming more mainstream. Streaming data and event processing has been part of the data landscape for many decades, but for much of that time, data streaming was a niche activity. Although adopted in industry segments with high-performance, real-time data processing and analytics requirements such as financial services and telecommunications, data streaming was far less common elsewhere. That has changed significantly in recent years, fueled by the proliferation of open-source and cloud-based streaming data and event technologies that have lowered the cost and technical barriers to developing new applications able to take advantage of data in-motion. This is a trend we expect to continue, to the extent that streaming data and event processing becomes an integral part of mainstream data-processing architectures.

Data from Ventana Research’s Analytics and Data Benchmark Research highlights both the growing use of streaming data and the reasons for that growth. Currently, less than one-quarter (21%) of organizations are in production with event-streaming technologies, with a further 16% planning to, and 18% still evaluating. Organizations utilizing event-streaming technologies have greater confidence in their ability to cope with the growing volume, variety and velocity of data. Almost three-quarters (73%) of organizations in production with event-streaming technologies are confident in their ability to analyze large volumes of data compared to 49% of those that are not using these technologies. Nearly nine-tenths (86%) of organizations in production with event-streaming technologies are confident in their ability to analyze the variety of data needed to make informed business decisions, compared to 65% of those that have not adopted event-streaming technologies. And almost two-thirds (64%) of organizations are confident in their ability to analyze high-velocity data while less than one-half (43%) of organizations who do not utilize event-streaming technologies are not. Ventana Research’s Streaming Data Dynamic Insights enables organizations to assess their relative maturity in achieving value from streaming data.

One reason the adoption of streaming data may be highly impactful is that it can be applied to both operational and analytic data workloads and business processes. To date, a slightly higher proportion of organizations have adopted streaming data for operational workloads. Adoption for analytic workloads is growing, which should soon equate to greater association with the generation of business intelligence and delivery of business value. Almost one-half (48%) of participants in Ventana Research’s Analytics and Data Benchmark Research are at organizations that use streaming data in operational processes, while 44% use streaming data in analytics processes. The use of streaming data in analytics processes is expected to accelerate, however, with 16% of organizations planning to use streaming data in analytics processes in the next 12 months, compared with 13% of organizations planning to use streaming data in operational processes in the same time frame. Streaming analytics — the use of technology to analyze information as it is generated in order to respond to opportunities or threats with timely actions — is becoming more mainstream, driven by the desire of organizations to increase the frequency of analysis to deliver more real-time decision-making, and the eagerness of business intelligence and visualization vendors to incorporate support for streaming analytics. My colleague, Dave Menninger, asserts that by 2025, 9 in ten business intelligence platforms will incorporate streaming analytics to mix historical analyses with real-time or near-real-time analyses.

Support for streaming analytics and real-time data processing is still a work in progress. Fewer than six in 10 (56%) organizations consider their analytics technologies for real-time analysis to be adequate, while less than one-half (49%) of organizations consider their technologies for event streaming to be adequate. Ongoing questions over the adequacy of event-streaming technology can be attributed the relative immaturity of products and services, as well as the historic tendency of organizations to deploy them for standalone, niche workloads. While vendors have focused on developing and refining core stream data processing and analytics functionality, there has been less focus on the management and governance of data in-motion alongside data at rest. This is changing, however. Established data management vendors added streaming and events technologies to their portfolios, and streaming and event specialists have added centralized management and governance capabilities — including data cataloging — to facilitate the discovery and governance of distributed data. Significantly, that means not just managing and governing streaming data in isolation, but embracing real-time data flows as the continuous generalization of batch data processes so that data in-motion and data at rest can be managed and governed holistically.

For this to become a reality, the capabilities that support the holistic management and governance of data in-motion and data at rest must be made available by the streaming data and event vendors, and data governance specialists. Almost two-fifths (39%) of participants in Ventana Research’s Data Governance Benchmark Research stated that a shortcoming of existing technologies for data governance is a failure to keep up with new data techniques and technologies (such as search, semantics, lineage and streaming data). I assert that by 2024, more than one-half of all organizations’ standard information architectures will include streaming data and event processing, allowing organizations to be more responsive and provide better customer experiences. I recommend that all organizations that have not already done so investigate the potential opportunities for streaming data and analytics in their organization, while those that have already adopted streaming data for niche workloads examine the potential for a more holistic approach to managing and governing data in-motion and data at rest.