What They Forgot to Teach You
Issue #3
👋 Welcome
I have done a lot of Writing Reviews over the last couple of months and there are a few patterns starting to emerge. Last issue I told you about the most common piece of advice I give: assume your reader knows nothing. The other problem that is more common than I wish it was, is not really about the writing. It is about the design and measurement decisions that were made months ago and how your marker might view them.
Here is a truth nobody tells you at the start of a research project: it is really difficult to write your way out of a design flaw. By the time you are writing up your thesis, you can try and justify the decision you made, but you can’t fix it. A thoughtful limitations section in your General Discussion shows your marker that you understand there was a problem, but the time to solve it was back when the study was a plan on the page.
In some programs, honours students have to write a proposal and then defend it in a panel meeting. My students used to dread this process, in part because it felt like a hurdle that was slowing them down, but also because it felt like it was too early in the process to have to defend their project. But it really was a gift. A design check, a feasibility check, a chance to get non-expert eyes on your study while you could still change it.
If your program doesn't build that kind of design check in, you can borrow the idea. Explain your design to someone outside your project. Have them ask you: What is your question? What do you plan to measure? WHY did you choose that task? What analysis will you use? What do you expect to happen? WHY do you predict it will turn out that way? Make sure you can answer these questions, before you collect data.
If you don't have a person to ask you those questions, I can be that person. I am running a small group coaching session around Experimental Design on August 7. We will work through the Questionable Measurement Practices questions in Flake and Fried's Measurement Schmeasurement paper. Details in the join me section below.
P.S. If you need design help sooner than that, book a Discovery call and we can chat.
📖 This week I read
Is high statistics anxiety as pervasive in university student populations as we think? — Loock & Counsell, The Score (APA Division 5)
Everybody "knows" psychology students are terrified of statistics, but is there empirical evidence for that idea? Johanna Loock and Alyssa Counsell (who runs a lab dedicated to statistics education research at Toronto Metropolitan University) did a meta-analysis of studies that had used the STARS scale (Statistics Anxiety Rating Scale), pooling data from 181 samples across 43 countries and testing whether statistics is really particularly scary to students.
The picture that emerges is more complicated than "everyone hates stats". Most students are actually a healthy level of anxious about statistics, the kind of anxious you'd expect to be about anything new and worth doing well. Johanna and Alyssa point out that these studies might also be overestimating stats anxiety because they tend to administer the STARS scale at the beginning of a course, when you actually don’t know what statistics is or what it involves.
But here's the thing. Even if levels of stats anxiety in students is lower than we have assumed, stats in the context of a real research project is still hard. But what if it was easy?
🎓 The hidden curriculum
What if stats was easy?
If most students are only a healthy level of anxious about stats, why does dealing with data in your thesis year feel so hard?
Here is my theory. Every stats course you have ever done has handed you a toy dataset. The data you work with is small, clean, complete, and carefully designed so that the ANOVA will "work" and the effect will be significant. Then you start your own project, and suddenly the data is yours. Your design, your messy spreadsheet, your missing values, your participant who answered 3 for every question. Nobody designed this dataset to behave. And that is the jump nobody warns you about. You are not doing stats exercises anymore, you are doing stats in the real world.
I have been rereading Oliver Burkeman's Meditations for Mortals (highly recommend this book btw), and on Day 15 the question he poses is "What if this were easy?" In this chapter, Burkeman talks about the false allure of effort, the idea that we tend to think that things worth doing should be hard so make them harder than they need to be. We treat the struggle as proof we are doing it properly. Students do this with data all the time. Hours of white-knuckled spreadsheet wrangling, wrestling with syntax that makes no sense, furrowing over output that doesn’t line up. What if it were easy?
What if it was about answering simple questions? What is my research question? Which test can help me answer my question? What does my data need to look like before I can run that test?
I have been playing with Burkeman's idea this week and building a couple of data tools with Claude, designed around the two places students get stuck most: getting your data from its raw state to an analysable state and choosing the right test for your question. What would those steps look like if they were easy?
I am looking for beta testers. If you want a sneak peek, let me know. I would really value your feedback.
💡 What I'm into
In related podcast listening, I have already recommended the Change, Technically podcast, and this week I went back to their episode "Math is for Girls". Here is an uncomfortable thought: your average psychology class is mostly women. So when we casually repeat "psych students are scared of stats", we are not saying something all that different from "girls are bad at math". In this episode, Cat and Ashley unpack the evidence behind that idea. Spoiler: it is not good. The anxiety is learned and it is passed on. Worth an hour of your commute.
📅 Join me
Small group coaching, N = 10, mini-masterclass followed by Ask Me Anything
Managing your time when no one is telling you what to do — July 17, 12–1:30pm AEST
You're managing a massive project, around a full time job, with no boss setting your deadlines and no colleagues to keep pace with.
Experimental Design Clinic — August 7, 12–1:30pm AEST
You cannot write your way out of a design flaw, but you can catch one while your study is still a plan on a page. We will work through Flake and Fried's six measurement questions as a group. Bring your design, borrow everyone else's eyes, and leave confident your plan will hold up.
Writing your General Discussion — August 21, 12–2pm AEST
You have your results. You know what you found. So why does the discussion feel like you just don't know where to start?
Register at www.revealresearch.org/events
🛠️ For your toolkit
Jenny's (very opinionated) tips for academic writing
I have pulled together 48 ideas about academic writing that you wish you knew sooner. This 12 page pdf contains tips about words, sentences, and paragraphs, for every section of your thesis. Specific, practical ideas that you can implement right now to make your writing sharper, clearer, and more confident. Get it at www.revealresearch.org
👋 thanks!!
Wishing you ease in your week…
Until next time, Jen
P.S. If you know someone feeling like stats is hard right now, forward this to them. It's free, and it might be exactly what they need.
P.P.S. If this is the first What They Forgot to Teach You newsletter to actually reach you, you might want to go back and find issues 1 and 2. They are probably in your spam. Yes, I am learning a lot about email filters.
I'm Jen, a developmental psychologist and former UNSW A/Prof. I help new researchers go from feeling like they've missed something everyone else knows, to feeling like they genuinely belong in research, so they can get through their research degree and into the career they actually want.
|
