3 Shifts in the Modern Data Environment and What it Means for IT Leaders | 7wData

• Read original article here

Providing organizations with reliable data for better decision-making is an undertaking that has not fundamentally changed in decades. Despite massive technology advances and new tactics, the IT organization managing data infrastructure today still has the same overall mission: moving data from its moment of creation and making it accessible and understandable by decision-makers at the moment of need.

However, while the objective has stayed the same, the obstacles to successfully create and maintain a source of analytical truth within a business have become exponentially more difficult.

Perhaps the biggest hurdle in recent years within the modern data environment has been new sources of data that generate unprecedented amounts of output, often with very little (if any) structure. From clickstreams, server logs, and social media sources to machine and sensor readings, the onslaught of data from these channels has been overwhelming—literally. From an economic and performance point of view, traditional enterprise data warehouses (EDWs) simply cannot keep up with this data tidal wave.

This has sparked a complete re-think of data capture and analysis strategies and given rise to a new generation of data storage solutions aimed at schema-less capture, hardware scalability, and the moving of compute capability closer to (if not on top of) data stores themselves.

Though still young by relational database standards, these newer, non-relational solutions have gained serious traction in recent years and matured rapidly to support some of the largest and most complex corporate enterprises in the world. While this has been done largely as a means to complement existing enterprise data warehouse infrastructures, it never the less creates a more complex data ecosystem for IT to manage.

Adding to the hurdles IT must overcome in the ongoing mission to maintain a healthy data environment is the availability of data from cloud applications. Many organizations use applications like Google Analytics, Salesforce, Netsuite, Zendesk, and others as core parts of their infrastructure.

The data they generate is critical to organizational reporting. Integrating data from these cloud solutions and making it accessible to the company has become a standard requirement for IT.

With the traditional EDW no longer functioning as the sole data destination, the question of “when, where, how, and if” to bring cloud application data into the corporate data environment is an ongoing and heated discussion.

Lastly, as self-service analytics for organizations of all sizes becomes the norm, more and more non-technical users (no formal IT/data training) are doing data discovery and reporting— sometimes even prep and advanced analysis. Businesses embracing this movement often see a dramatic reduction in (if not full elimination of) IT’s responsibilities for producing analytics.

While this shift is critical to the overall success of an organization adopting a data-driven mindset, it puts new pressure on IT groups to provide broader data access. All of this is in addition to ensuring the technology satisfies business needs while meeting IT’s requirements for security and governance.

In an effort to meet these new challenges, many IT organizations rush to adopt new technologies and tactics but fail to see how these hurdles have actually shifted the way IT groups need to approach the goal of managing data from “creation to consumption.” Big Data solutions, cloud data integration, and self-service analytics are all answers to bigger technology problems, but in order to deploy them effectively in an organization, the IT playbook needs to change.

This article seeks to outline three major thought shifts concerning the modern data environment that IT leaders need to understand in order to support data-driven decision making within their organization.

The enterprise data warehouse is not dead. It just has more friends.