The big idea: should we be using data to make life’s big decisions?

• Read original article here

Whom should you marry? Where should you live? How should you spend your time? For centuries, people have relied on their gut instincts to figure out the answers to these life-changing questions. Now, though, there is a better way. We are living through a data explosion, as vast amounts of information about all aspects of human behaviour have become more and more accessible. We can use this big data to help determine the best course to chart.

There has long been overwhelming – and often surprising – evidence that algorithms can be much better than people at making difficult decisions. Researchers have collected data on various kinds of choices people make, the information they base those choices on, and how things turn out. They have found, for example, that a simple data-driven algorithm would have been better than judges at deciding whether a defendant should stay in jail or be released; better than doctors at deciding whether a patient should get a procedure; and better than school principals at deciding which teachers should be promoted.

The power of data analysis has been proved in the sports and business worlds, too. As made famous by the book and movie Moneyball, baseball teams found that algorithms were better than scouts at picking players, and better than managers at picking strategies. In finance, the hedge fund Renaissance Technologies dramatically outperformed competitors by seeking out patterns in stock market data and using them to inform its investment strategy. Tech firms in Silicon Valley have found that data from experiments provides better insights into how to design their websites than designers could.

But stats have had surprisingly little impact, thus far, on our personal lives. One major problem is that good data about life’s biggest personal questions has been difficult to come by. The revolution may have come to baseball early thanks to all the information about performance that its obsessive fans had demanded and collected. Now we can anticipate a “Lifeball” moment as a result of all the data that our smartphones and computers are able to harvest.

Consider this not-too-trivial question: what makes people happy? Data to answer this question in a rigorous, systematic way was simply not available in the 20th century. While play-by-plays from every game provided raw material for data scientists working in sports, there was no equivalent record of events in people’s lives and the changes and mood they provoked. Happiness, unlike baseball, was simply not open to quantitative research

It is now. Experience sampling projects ping people on their devices and ask them various questions: what are you doing? Who are you with? How happy are you? The largest of these, Mappiness, co-founded by the UK-based economists Susana Mourato and George MacKerron, has collected a repository of more than 3m data points. They have revealed the activities that provide far more enjoyment than most of us would have guessed, such as exercising, going to a museum and gardening. Then there are the things that give us less pleasure than you might assume, such as playing video games, watching TV and browsing the internet. Watching sports fixtures involving your favourite team can be particularly dangerous to your mood, it turns out. The average sports fan gets 3.9 points of happiness when their team wins, but forfeits 7.8 points of happiness when it loses.

There is even some evidence that just telling people about the data on happiness can increase it. One randomised controlled trial found that indicating which activities had been found most likely to bring them pleasure, combined with a plan to incorporate more of them into daily life, led to improved mood.

Another way to be happier is to marry well. Here, too, data is offering us new insights. One study by 86 researchers collected information on more than 11,000 romantic couples. They used machine learning models to understand what predicts romantic satisfaction. They found many highly desired traits, such as a partner’s attractiveness and height, have just about no correlation with long-term happiness. Instead the qualities most predictive of romantic satisfaction tended to be psychological ones, such as having a so-called “growth mindset”, or a secure attachment style.

And one final data-driven strategy for happiness is moving house. A study by three economists at the National Bureau of Economic Research in Cambridge Massachusetts analysed survey data and ranked the happiness of every corner of the US. They found that when people moved from an unhappy city to a happier place, the effect rubbed off on them, and their overall mood improved.

Clearly, data based on large samples of people isn’t all you need consider when moving. An individual might not want to pack up and head to Charlottesville, Virginia – the happiest place in the US – based on these surveys alone. Interestingly, data is giving us insights into many of the other factors that might come into play. For example, a study of tens of millions of children has found the places that increase their future earnings the most. Another with a huge sample size found that certain cities can improve one’s life expectancy.

These are the early days of the data revolution in personal decision-making. I am not claiming that we can completely outsource our lifestyle choices to algorithms, though we might get to that point in the future. I am claiming instead that we can all dramatically improve our decision-making by consulting evidence mined from thousands or millions of people who faced dilemmas similar to ours. And we can do that now.

Don’t Trust Your Gut: Using Data to Get What You Really Want in Life by Seth Stephens-Davidowitz is published by Bloomsbury.