When plotting all the data can help avoid overinterpretation

• Read original article here

Statistical Modeling, Causal Inference, and Social Science
When plotting all the data can help avoid overinterpretation
Posted on
Patrick Ruffini and  David Weakliem both looked into this plot that’s been making the rounds, which seems to suggest a sudden drop in some traditional values:
But the survey format changed between 2019 and 2023, both moving online and randomizing the order of response options.
Perhaps one clue that you shouldn’t draw sweeping conclusions specific to these values is that there is a drop in the importance of “self-fulfillment” and “tolerance” too. Weakliem writes that once you collapse a couple response options…
there’s little change–they are almost universally regarded as important at all three times. The results for “self-fulfillment,” which isn’t mentioned in the WSJ article, are particularly interesting–the percent rating it as very important fell from 64% in 2019 to 53% in 2023. That’s hard to square with either the growing selfishness or the social desirability interpretations, but is consistent with my hypothesis. These figures indicate some changes in the last few years, but not the general collapse of values that is being claimed.
If the importance of everything drops at once, this might be a clue that selective interpretation of some thematically-related drops is likely not justified — whether this is because of survey format changes or otherwise (say something else becoming comparatively more important, but not asked about).
So perhaps this is a good reminder of the benefits of plotting more of the data — even if you want to argue the action is all in a few of the items. (You could even think of this as something like a non-equivalent comparison group or differences-in-differences design.)
[This post is by Dean Eckles.]
This entry was posted in Miscellaneous Statistics , Sociology , Statistical graphics by Dean Eckles . Bookmark the permalink .
Leave a Reply
Your email address will not be published. Required fields are marked *
Comment *
Raghu Parthasarathy on Recent book reviews
April 3, 2023 10:35 AM
Apparently I'm two volumes behind on "The Arab of the Future"! I thought I must have missed the English translation…
chipmunk on An epidemiologist and a psychologist come down from the ivory tower to lecture us on “concrete values like freedom and equality” . . . What does that even mean?? The challenge of criticizing policies without considering their political and social contexts:
April 3, 2023 10:19 AM
Dale: I don't see the COVID policy critics as great visionaries, but they had some valid points that were shouted…
April 3, 2023 10:10 AM
Typo reglé; merci.
April 3, 2023 9:40 AM
Thanks for the telegraphic reviews, mais on dit L'Arabe du Futur en français...!
Daniel Lakeland on The behavioral economists’ researcher degree of freedom
April 3, 2023 6:53 AM
Thomas, yes of course, Bayesian probability, which is not a property of the physics of the basketball shot, may vary…
David in Tokyo on The behavioral economists’ researcher degree of freedom
April 3, 2023 5:34 AM
Daniel L. wrote: "When the basketball player takes his Nth shot, that is a sequence of actions which is unique,…
Anonymous on An epidemiologist and a psychologist come down from the ivory tower to lecture us on “concrete values like freedom and equality” . . . What does that even mean?? The challenge of criticizing policies without considering their political and social contexts:
April 3, 2023 5:04 AM
What I am starting to wonder more and more is whether (some) scientists think they have the right to nudge…
Ewout Steyerberg on “Risk ratio, odds ratio, risk difference… Which causal measure is easier to generalize?”
April 3, 2023 3:34 AM
Confused about “ So it’s not possible to make the “quartet” plots with raw data”. The graphs condition on a…
Thomas on The behavioral economists’ researcher degree of freedom
April 3, 2023 2:30 AM
apologies! i wrote a comment which now I cannot see, and then added a correction... The comment was something like…
Thomas on The behavioral economists’ researcher degree of freedom
April 3, 2023 2:22 AM
(esprit d'escalier...) - I should have said why can't we allow the phenomena which give rise to a probability assessment…
Thomas on The behavioral economists’ researcher degree of freedom
April 3, 2023 2:12 AM
Ah, I see, you say probability does not exist in general, not just as regards apes shooting basketballs. Sure, in…
Eric Rasmusen on “Risk ratio, odds ratio, risk difference… Which causal measure is easier to generalize?”
April 2, 2023 4:58 PM
In your paper, you might note that 2e is the linear-no-threshold for harm from a treatment, and 2f is a…
Eric Bennett Rasmusen on “Risk ratio, odds ratio, risk difference… Which causal measure is easier to generalize?”
April 2, 2023 4:28 PM
I thought this was going to be a post on which summary statistic to use, another good topic (would be…
Jon on “Risk ratio, odds ratio, risk difference… Which causal measure is easier to generalize?”
April 2, 2023 2:43 PM
There is not a single new result on this paper... Morever, to give credit to Anders, he is selling himself…
D.O. on The behavioral economists’ researcher degree of freedom
April 2, 2023 12:05 PM
People tend to believe what their peers believe. This applies both to basketball coaches and academics. But mavericks and contrarians,…
Chris Wilson on “Risk ratio, odds ratio, risk difference… Which causal measure is easier to generalize?”
April 2, 2023 10:54 AM
The default assumption of a constant effect is also just a convenient computational shortcut in many cases - plug in…
Daniel L Speyer on The behavioral economists’ researcher degree of freedom
March 31, 2023 4:38 PM
The Two Models of Microecon thing is very clear if you've read [Inadequate Equilibria](https://equilibriabook.com/toc/). If people with decent visibility on…
Daniel L Speyer on The behavioral economists’ researcher degree of freedom
March 31, 2023 4:35 PM
The observed effect could be explained by either model. Distinguishing between them might be possible by looking at correctly-aimed-but-blocked shots,…
JYD on The behavioral economists’ researcher degree of freedom
March 31, 2023 4:22 PM
“ They seem strongly committed to the idea that basketball players can’t be meaningfully influenced by previous shots” You seem…
Jonathan (another one) on The behavioral economists’ researcher degree of freedom
March 31, 2023 1:14 PM
A classic example here is punting on fourth down in football. Coaches learned to do so, and doing so was…
GabbyD on The behavioral economists’ researcher degree of freedom
March 31, 2023 1:04 PM
Do people have money on the line on Hot Hand? Is this the moneyball concept? Can teams really profit out…
Jonathan (another one) on The behavioral economists’ researcher degree of freedom
March 31, 2023 12:50 PM
Humans are a thin stratum of rationality on top of a large mass of habit. But the habits come from…
L&M on The behavioral economists’ researcher degree of freedom
March 31, 2023 12:20 PM
Part of the "hot hand fallacy" could be philosophical differences in randomness. Coin flips are usually considered random but if…