Top 10 Big Data Challenges and How to Address Them | 7wData

• Read original article here

A well-executed big data strategy can streamline operational costs, reduce time to market and enable new products. But enterprises face a variety of big data challenges in moving initiatives from boardroom discussions to practices that work.

IT and data professionals need to build out the physical infrastructure for moving data from different sources and between multiple applications. They also need to meet requirements for performance, scalability, timeliness, security and data governance. In addition, implementation costs must be considered upfront, as they can quickly spiral out of control.

Perhaps most importantly, enterprises need to figure out how and why big data matters to their business in the first place.

"One of the greatest challenges around big data projects comes down to successfully applying the insights captured," said Bill Szybillo, business intelligence manager at ERP software provider VAI.

Many applications and systems capture data, he explained, but organizations often struggle to understand what is valuable and, from there, to apply those insights in an impactful way.

Taking a broader look, here are 10 big data challenges that enterprises should be aware of and some pointers on how to address them.

Big data by its very definition typically involves large volumes of data housed in disparate systems and platforms. Szybillo said the first challenge for enterprises is consolidating the extremely large data sets they're extracting from CRM and ERP systems and other data sources into a unified and manageable big data architecture. Once you have a sense of the data that's being collected, it becomes easier to narrow in on insights by making small adjustments, he said. To enable that, plan for an infrastructure that allows for incremental changes. Attempting big changes may just end up creating new problems. To shepherd a big data initiative from boardroom discussions to business insights, enterprises face a number of challenges.

The analytics algorithms and artificial intelligence applications built on big data can generate bad results when data quality issues creep into big data systems. These problems can become more significant and harder to audit as data management and analytics teams attempt to pull in more and different types of data. Bunddler, an online marketplace for finding web shopping assistants who help people buy products and arrange shipments, experienced these problems firsthand as it scaled to 500,000 customers. A key growth driver for the company was the use of big data to provide a highly personalized experience, reveal upselling opportunities and monitor new trends. Effective data quality management was a key concern. "You need to monitor and fix any data quality issues constantly," Bunddler CEO Pavel Kovalenko said. Duplicate entries and typos are common, he said, especially when data comes from different sources. To ensure the quality of the data they collect, Kovalenko's team created an intelligent data identifier that matches duplicates with minor data variances and reports any possible typos. That has improved the accuracy of the business insights generated by analyzing the data.

Big data platforms solve the problem of collecting and storing large amounts of data of different types -- and the quick retrieval of data that's needed for analytics uses. But the data collection process can still be very challenging, said Rosaria Silipo, a Ph.D. and principal data scientist at open source analytics platform vendor Knime. The integrity of an enterprise's collected data stores is dependent on them being constantly updated. This requires maintaining access to a variety of data sources and having dedicated big data integration strategies. Some enterprises use a data lake as a catch-all repository for sets of big data collected from diverse sources, without thinking through how the disparate data will be integrated.