Do Your Data Scientists Know the ‘Why’ Behind Their Work? | 7wData

• Read original article here

data science has been around for a long time. But the failure rates of big data projects and AI projects remain disturbingly high. And despite the hype, companies have yet to cite the contributions of data science to their bottom lines.

Why is this the case? In many companies, data scientists are not engaging in enough of softer, but more difficult, work, including gaining a deep understanding of business problems; building the trust of decision makers; explaining results in simple, powerful ways; and working patiently to address concerns among those impacted.

Managers must do four things to get more from their data science programs? First, clarify your business objectives and measure progress toward them. Second, hire data scientists best suited to the problems you face and immerse them in the day-in, day-out work of your organization. Third, demand that data scientists take end-to-end accountability for their work. Finally, insist that data scientists teach others, both inside their departments and across the company.

Data science, broadly defined, has been around for a long time. But the failure rates of big data projects in general and AI projects in particular remain disturbingly high. And despite the hype (e.g., “data is the new oil”), companies have yet to cite the contributions of data science to their bottom lines. What is going on?

Recently, Ron Kenett, the distinguished Israel-based data scientist, and I compared notes on our own successes and failures — and those of our colleagues — in helping companies with data science. It was immediately clear that the biggest successes stemmed not simply from technical excellence but from softer factors such as a deep understanding of business problems; building the trust of decision makers; explaining results in simple, powerful ways; and working patiently to address dozens of concerns among those impacted. Conversely, otherwise excellent technical work died on the vine when we failed to connect with the right people, at the right times, or in the right ways.

In many companies, data scientists are not engaging in enough of this softer, but more difficult, work. Two underlying reasons contribute. First, many data scientists are much more interested in pursuing their crafts — namely, finding interesting nuggets buried in data — than they are in solving business problems. In some respects, this is natural. After all, they are taught a narrow focus on data and the tools needed to explore it, and doing so helps them earn peer recognition. Plus, applying advanced techniques is more fun than dealing with the messy realities of corporate life.

The second reason: From the company’s perspective, the talent is rare and protecting data scientists from the chaos of everyday work just makes sense. But doing so increases the distance between data scientists and the company’s most important problems and opportunities. Exacerbating this, for many organizations, data scientists are new and unfamiliar, and companies are still learning how to manage them. It is tempting to bolt data science onto your existing organization and hope for the best.

So, what should managers do to get more from their data science programs?

First, clarify your business objectives and measure progress toward them. While data science does require initial investment, you should expect real results — in terms of cost savings, new revenue, improved customer satisfaction, or risk reduction — within a couple of years. Obvious as this sounds, the implications are profound. For most, it means recognizing that you are not ready for overhyped technologies, such as machine learning, and focusing first on more basic opportunities, such as putting operational processes under control, improving data quality, and developing a deeper understanding of customers.

Second, hire data scientists best suited to the problems you face and immerse them in the day-in, day-out work of your organization.