Episode Details

English Podcast Starts at 00:00:00

Bengali Podcast Starts at 00:16:27

Hindi Podcast Starts at 00:32:01

Danish Podcast Starts at 00:45:38

Reference

Kay, C.S. Why you shouldn’t trust data collected on MTurk. Behav Res 57, 340 (2025). https://doi.org/10.3758/s13428-025-02852-7

The materials, data, and analytic code for the four studies can be found on OSF (https://osf.io/frwq4/?view_only=2e9d981a2df1411d8971a120ae75df06). All four studies were preregistered. The preregistration for the Connect study can be found on OSF (https://osf.io/wzf5u/?view_only=8c300b94bfe544948901bd4028370b35). The preregistration for the Prolific study can be found on OSF (https://osf.io/2g4ry/?view_only=9f4760a0e30c444b890408d256c5da84). The preregistration for the open MTurk study can be found on OSF (https://osf.io/wzf5u/?view_only=8c300b94bfe544948901bd4028370b35). The preregistration for the qualified MTurk study can be found on OSF (https://osf.io/8rxz7/?view_only=15e64202d55e4df183b44b3517f60083)

⁠

Youtube Channel

⁠https://www.youtube.com/@weekendresearcher⁠

Connect over linkedin

https://www.linkedin.com/in/mayukhpsm/

Love data? Love clean methods? Love that tiny thrill when a correlation points the right way? Then welcome to Revise and Resubmit 🎙️✨

Today’s episode begins with a simple belief most of us carry like a lab notebook in our heads: if you ask people two opposite questions, their answers should pull apart. “I talk a lot” should not walk hand in hand with “I rarely talk.” They should disagree. They should argue. They should, at the very least, behave like opposites in the wild.

But what if they don’t? What if they lock arms and march in the same direction like they’ve rehearsed it? 😬📉

In a crisp, almost disarming demonstration, researcher Cameron S. Kay takes 27 semantic antonym pairs and runs them across three crowdsourcing worlds: Connect, Prolific, and MTurk. On Connect and Prolific, reality behaves. Opposites repel, correlations lean negative, language keeps its promises. On MTurk, though, something eerie happens: over 96%96% of those “opposites” move together, positively correlated, like contradiction has quietly been rewritten into compliance 🤖🧪

Even the usual defenses don’t work. Attention checks? Still broken. Filtering for high-reputation, high-productivity workers? Still broken. The kind of screening we lean on when we want to sleep at night? Still broken. And the implication lands with a thud: data from MTurk may not just be noisy, it may be fundamentally untrustworthy for behavioral research 🔍⚠️

This paper appears in Behavior Research Methods, a prestigious Scopus Q1Q1 journal, published on 10 November 2025 (Volume 57, article number 340), and it reads like a wake-up call that’s polite enough to invite you in, then firm enough to not let you look away 📚🏛️

Before we dive in, do the good podcast-citizen thing: subscribe to “Revise and Resubmit” on Spotify 🎧, and catch the video side on YouTube at “Weekend Researcher” ▶️. You can also find us on Amazon Prime and Apple Podcast 🍏📺

And a sincere thank you to Cameron S. Kay and to Springer Nature for bringing this research into the world 🙏📄

So here’s the question that lingers in the doorway, waiting for you to answer it: if a platform can make contradictions agree, what else might it be quietly teaching our science to believe? 🤔💭