What They Forgot to Teach You
Issue #7
๐ Welcome
I have read a lot of results sections over the last few weeks, and something keeps jumping out at me. Or rather, something keeps NOT jumping out at me.
The figures.
Draft after draft, I get a results section that is tables all the way down. A table of means and standard deviations. A table of correlations. A neat block of regression output pasted straight from the software. I can count on one hand the number of theses I have reviewed this year that include a plot of the actual data.
Now, a small confession. At first I told myself this was fair enough. Most of the theses I read these days are survey based, questions about whether one thing predicts another, a mediation here, a moderation there. That is a different beast from the experimental questions I spent years marking at UNSW, where you have conditions to compare and an obvious difference to plot. Maybe, I thought, correlational projects just donโt have much to look at.
Then I did some reading and I changed my mind. Even when your question is about the relationship between two variables, there are still MANY reasons to plot that relationship, for you and for your reader. If anything, a survey project needs more looking, not less.
Plotting is not the thing you do at the end to pretty up your results. It is the thing you do as soon as you have your data, so you actually know what is going on in it.
๐ This week I read
Same Stats, Different Graphs โ Justin Matejka & George Fitzmaurice, Autodesk Research (2017). Read it here โโ
Matejka and Fitzmaurice built a set of 13 datasets they call the Datasaurus Dozen. Much like the classic Anscombe quartet, every one of them has the same mean, the same standard deviation, and the same correlation, matched to two decimal places.
โThen you plot them. One is a cluster of lines. One is a star. One is a circle. And one is a dinosaur. You can access these datasets using the datasauRus R package.
The takeaway here is that descriptive stats hide important information. If you rely on the numbers alone, you will never find the dinosaur hiding in your spreadsheet!
๐ The hidden curriculum
Let me show you what I mean, with a study story of my own. Years ago I was working on a study using EMG to measure facial mimicry in infants. We were recording tiny electrical signals from two muscles: the one in your cheek that lifts when you smile, and the one in your brow that pulls in when you frown. The plan was to see whether babies mimicked happy and angry faces by activating the corresponding muscle. For the longest time, it looked like for angry faces there was nothing going on.
It was only when I plotted the raw muscle activity that I saw what was going on. As it turns out, the brow muscle sits at a much higher baseline than the cheek muscle, just resting, between trials, when there is nothing on the screen. The standard way of scoring the data assumed both muscles started from the same low baseline, which made it much harder to ever see mimicry of the angry faces, because the brow signal started so high there was less room left for it to climb. My lack of effect turned out to be a measurement problem, rather than a baby problem.
My mimicry study is an example of how plotting can help you understand your data after you have run some stats, but plotting your data before you even think about stats also pays off. A quick histogram of every variable will help you catch the participant who entered their age as 999 and the wellbeing item you forgot to reverse-score. It is also where you find out whether your data actually meets the assumptions of the test you had in mind. Is the outcome variable roughly normal or piled up at one end? Is the relationship between x and y a straight line or does it bend? Sometimes the plot tells you the test you planned is the wrong one, while you can still change your mind.
Plotting earns its keep most at the stage everyone dreads, when the effect you hoped for comes back non-significant. A scatterplot can show you that almost everyone scored near the top of your stress scale (a ceiling, not a null result), or that a single unusual participant is dragging your regression line around, or that your two groups overlap so heavily there was never much of a difference to find. A plot can also tell you that despite there not being a significant difference between your groups, there is an interesting difference in the distribution of scores across groups. You might have a Datasaurus problem: means might look the same but the pattern of data is still something you can tell a story about.
And it keeps paying off when you sit down to write. This is really what people mean by exploratory data analysis: looking at your data with an open mind and letting it show you things that you did not anticipate. I love reading a discussion where the student has a hunch about why their experiment didn't pan out, finds some evidence of that explanation in the literature AND goes back to their data to show that there is actually support for that explanation in their data, when it is plotted in a way they hadn't planned.
Finally, a good figure is also how you make the story of your analysis land for a tired reader/marker. A plot carries your result more convincingly than a table of coefficients, because your marker can see the effect rather than having to reconstruct it from a stack of numbers in a table. Showing the raw data alongside the test statistic (not instead of it) is increasingly what journals and reviewers expect, but really it is just doing your reader a kindness.
Before you run your analysis, plot everything. A histogram of every variable, the scatterplot behind every correlation. If you are not sure how to make a start on that challenge, join us for new plotting sessions (register below).
๐ก What I'm into
Lately I have been into the Data Science Hangout and the Data Science Lab YouTube channels. If you are just starting to dip your toes into R, they are a great place to find inspiration. Posit runs these community events live at a time that is ridiculously incompatible with life in this hemisphere, but they always land on YouTube within a few weeks, so you can watch on your own schedule.
A good place to start is Cara Thompsonโs episode. She is a psychology PhD turned data visualisation consultant, whose whole thing is turning messy data into something that helps your reader understand your story.
๐
Join me
Free Friday workshops โ one hour, Zoom session, leave with tips and tricks
- How to write a lot when you have no time โ Friday August 28, 12pm AEST. Your thesis gets done in the small pockets you carve out and protect, not the clear week that never comes. Includes a live "Shut Up and Write".
- How to polish your way to a High Distinction โ Friday September 4, 12pm AEST. Your ideas are good, the HD is in how easily your tired marker can grasp them.
Small group coaching โ N= 5, two hours, Zoom session, bring your results and/or data and work through outlining your discussion or making compelling plots together.
- Keep Calm and Write Your General Discussion โ Fridays August 21, September 11 or September 25, 12-2pm AEST. N = 5, 2 hours, A$49. You have your results, you just don't know where to start. We work through the six discussion moves together and you leave with an outline.
- **NEW: Plotting for survey research** โ Fridays September 4, September 18 or October 2, 1-3pm AEST. N = 5, 2 hours, A$49. Think plotting is only for experiments with neat conditions to compare? Survey data needs more looking, not less. Bring your data and Iโll help you build the figures that check for problems and tell your story.
Register for everything at www.revealresearch.org/events.
๐ ๏ธ For your toolkit
9 reasons to plot your data. I pulled every reason plotting is worth it into a one-page handout, organised by where in the process it saves you. It's the companion to the plotting for survey research sessions, and might be useful as you begin to plot your data. Grab it here โ
๐ thanks!!
So this week, before you start your analysis, please plot your data. Worst case, you make some pretty pictures. Best case, you find the dinosaur before your marker does.
Until next time, Jen
P.S. If a friend's results section is tables all the way down, forward this. It's free, and it might be exactly what they need.
P.P.S. If this is the first What They Forgot to Teach You to actually reach you, issues 1 through 6 are almost certainly sulking in your spam folder. Go and rescue them.
I'm Jen, a developmental psychologist and former UNSW A/Prof. I help new researchers go from feeling like they've missed something everyone else knows, to feeling like they genuinely belong in research, so they can get through their research degree and into the career they actually want.
|
โ