Don’t rely too much on social media to understand human behavior
Social media can give us some insights into human behavior but it can also give us a lot of nonsense
Absurd science
The motivation for our paper comes from a series of research studies that deliberately present absurd scientific results.
One brain imaging studyappeared to show the neural activity of a dead salmon tasked with identifying emotions in photos. Ananalysis of longitudinal statistics from public health recordssuggested that acne, height, and headaches are contagious. And ananalysis of human decision-makingseemingly indicated people can accurately judge the population size of different cities by ranking them in alphabetical order.
Why would a researcher go out of their way to explore such ridiculous ideas? The value of these studies is not in presenting a new substantive finding. No serious researcher would argue, for example, that a dead salmon has a perspective on emotions in photos.
Rather, the nonsensical results highlight problems with the methods used to achieve them. Our research explores whether the same problems can afflict studies that use data from social media. And we discovered that indeed they do.
Positive and negative results
When a researcher seeks to address a research question, the method they use should be able to do two things:
For example, imagine you have chronic back pain and you take a medical test to find its cause. The test identifies a misaligned disc in your spine. This finding might be important and inform a treatment plan.
However, if you then discover the same test identifies this misaligned disc in a large proportion of the population who do not have chronic back pain, the finding becomes far less informative for you.
The fact the test fails to identify a relevant, distinguishing feature of negative cases (no back pain) from positive cases (back pain) does not mean the misaligned disc in your spine is non-existent. This part of the finding is as “real” as any finding. Yet the failure means the result is not useful: “evidence” that is as likely to be found when there is a meaningful effect (in this case, back pain) as when there is none is simply not diagnostic, and, as result, such evidence is uninformative.
XYZ contagion
Using the same rationale, we evaluated commonly used methods for analyzing social media data — called “null hypothesis significance testing” and “correlational statistics” — by asking an absurd research question.
Past and current studies have tried to identify what factors influence Twitter users’ decisions to retweet other tweets. This is interesting both as a window into human thought and because resharing posts is a key mechanism by which messages are amplified or spread on social media.
So we decided to analyze Twitter data using the above standard methods to see whether a nonsensical effect we call “XYZ contagion” influences retweets. Specifically, we asked
Upon analyzing six datasets containing hundreds of thousands of tweets, the “answer” we found was yes. For example, in a dataset of 172,697 tweets about COVID-19, the presence of an X, Y, or Z in a tweet appeared to increase the message’s reach by a factor of 8%.
Needless to say, we do not believe the presence of Xs, Ys, and Zs is a central factor in whether people choose to retweet a message on Twitter.
However, like the medical test for diagnosing back pain, our finding shows that sometimes, methods for social media data analysis can “reveal” effects where there should be none. This raises questions about how meaningful and informative results obtained by applying current social science methods to social media data really are.
As researchers continue to analyze social media data and identify factors that shape the evolution of public opinion, hijack our attention, or otherwise explain our behavior, we should think critically about the methods underlying such findings and reconsider what we can learn from them.
What is a ‘meaningful’ finding?
The issues raised in our paper are not new, and there are indeed many research practices that have been developed to ensure results are meaningful and robust.
For example, researchers are encouraged to pre-register their hypotheses and analysis plans before starting a study to prevent a kind of data cherry-picking called“p-hacking”. Another helpful practice is to check whether results are stable after removing outliers and controlling forcovariates. Also important arereplication studies, which assess whether the results obtained in an experiment can be found again when the experiment is repeated under similar conditions.
These practices are important, but they alone are not sufficient to deal with the problem we identify. While developing standardized research practices is needed, the research community must first think critically about what makes a finding in social media data meaningful.
Article byJason Burton, PhD researcher,Birkbeck, University of London;Nicole Cruz, Postdoctoral Research Associate,UNSW, andUlrike Hahn, Professor of Psychology,Birkbeck, University of London
This article is republished fromThe Conversationunder a Creative Commons license. Read theoriginal article.
Story byThe Conversation
An independent news and commentary website produced by academics and journalists.An independent news and commentary website produced by academics and journalists.
Get the TNW newsletter
Get the most important tech news in your inbox each week.
