The Data Daily

How much evidence do we need for a data visualization "rule"?

Last updated: 05-16-2019

Read original article here

How much evidence do we need for a data visualization

Thursday, June 7, 2018
How much evidence do we need for a data visualization "rule"?
In a separate post , I laid out some of my arguments for why I think most line charts should start at zero. I posted some of my initial thoughts on that topic on Twitter, which generated some really thoughtful replies.
One of them, from Steve Haroz, noted that he knew of know evidence that people read non-zero-baseline bar charts any differently than non-zero-baseline line charts. And, furthermore, that we should be careful in talking about data visualization "rules" when our evidence for them is weak or nonexistent.
Rely on evidence. When there is little or none, don't claim that some absolute or universal rule exists. It's fine to say "maybe" or "I prefer".
— Steve Haroz (@sharoz) May 24, 2018
This led to a quite spirited discussion about whether data-visualization "guidelines" or "rules of thumb" that don't have any empirical research to back them up can still be valuable, or if we should stick primarily to those things that we have solid evidence for.
Speaking personally, I didn't fully appreciate the gaps in data visualization research until I watched Robert Kosara's excellent talk at the University of Washington , "How Do We Know That?"
The talk is based on Kosara's paper, Empire of Sand , which I now assign to my students at the University of Florida.
As Kosara points out, many of the things we think we know about data visualization have little empirical evidence to back them up. And other well-accepted "rules" may actually be wrong (for example, "chartjunk" may not be so bad after all ).
Some rules are based on nothing more than the strong opinions of influential early writers in the field (like Edward Tufte  and Jacques Bertin ) and have not actually been subject to peer-reviewed research.
So where does that leave us as data visualization practitioners and teachers?
It would seem obvious that we shouldn't teach "rules" that we know to be wrong. But what about the many areas for which there is little or no empirical evidence at all? Can theory replace research in some cases? Is a common practice worth teaching our students even if we don't know it to be true?
Below, I've tried to collect some of my own thoughts on the matter as well as those of others who took part in the Twitter discussion.
First, though, a big caveat about my own tweets: While I teach at a university and have (strong) opinions on how to teach data visualization, I'm an "instructor" not a "professor". I don't have a PhD and I'm not engaged in academic research myself.
Let's get to the tweets!
My argument is based more on encoding consistency, though, and prudence: if encoding is height, then it may be safer if height is proportional to numbers
— Alberto Cairo (@albertocairo) May 24, 2018
We don't know what visual cue people use or how they reason with it. Could be length. Could be area. Could be position of the top of the bar. And not clear why a point's distance from the x-axis is different from the length of a bar. There are plausible options, but no evidence.
— Steve Haroz (@sharoz) May 24, 2018
That’s the key point! There’s elegance, Bertin-style retinal variables, etc., and then there’s the actual psychology, cognition, etc., and there are still big gaps in our knowledge of the latter.
— Robert Kosara ???? (@eagereyes) May 24, 2018
Why give credence to the former?
— Steve Haroz (@sharoz) May 24, 2018
Cause we don’t have enough of the latter?
— Robert Kosara ???? (@eagereyes) May 24, 2018
Medicine doesn't know how to cure every cancer. That doesn't mean that shamans and witch doctors are worth anyone's time.
— Steve Haroz (@sharoz) May 24, 2018
It strikes me that both these statements can be true:
1. We need more research on how people interpret charts to ensure our guidelines are backed by actual evidence.
2. In the meantime, guidelines can still be helpful, even if they're not based on peer-reviewed research
— Chad Skelton (@chadskelton) May 24, 2018
I'd say that would be a big improvement over the field 5-10 years ago, which basically was 1. We need to always do what it says to do in a couple really popular books on this topic, and 2. Anyone who doesn't is an idiot.
— Ben Jones (@DataRemixed) May 24, 2018
I think the key, which to be fair I think @sharoz agrees with, is that we should be upfront about the nature of our advice:
- This is backed by peer-reviewed research
- This is based on best practices in the field
- This is based on a hunch I had this morning
— Chad Skelton (@chadskelton) May 24, 2018
I agree it's good advice. Just call it "common practice" instead of best.
— Steve Haroz (@sharoz) May 24, 2018
I tend to be a bit conservative ideologically, at least in some areas. I think that traditions and customs often have good reasons to exist (until they are solidly refuted)
— Alberto Cairo (@albertocairo) May 24, 2018
Tufte has strong opinions that are largely unburdened by evidence. That’s a huge problem for the field. Bertin at least gave us a good framework to think in, even if some of his ideas might prove to be wrong.
— Robert Kosara ???? (@eagereyes) May 24, 2018
I value internal consistency (more visible in Bertin's work) and how it translates into a style. The more it overlaps scientific evidence the better, but it's not a requirement.
Heresy, I know.
— Jorge Camoes (@wisevis) May 24, 2018
I like @chadskelton 's "have it both ways" framing - there's value in both. Attempts to reason about best practices based on what we think we understand now are useful since if we waited until we had empirical studies answering every question, we'd wait a zillion years!
— Tamara Munzner (@tamaramunzner) May 25, 2018
Of course empirical studies are worth doing to refine our understanding. But the combinatorial explosion of questions will outstrip the number of answers, especially since every answer leads to so many followup questions. We'll never be done. So we continue to need both.
— Tamara Munzner (@tamaramunzner) May 25, 2018
Hmm. Maybe a rating & review system for rules of thumb would be interesting. X% of practitioners follow Rule A virtually always, Y follow it often, Z follow it rarely, etc, with comments & links to examples & studies...
— Ben Jones (@DataRemixed) May 25, 2018
Great idea! Sounds similar to the University of Chicago economist survey. Interestingly, that survey is *not* anonymous, which I think makes it better as people have to stand by their views. https://t.co/CjyD8sqoHC
— Chad Skelton (@chadskelton) May 25, 2018
I believe Min Chen at Oxford is doing something along these lines ...

Read the rest of this article here