Subscribe to RSS
Australasian Institute of Digital Health Summit 2022–Automated Social Media Surveillance for Detection of Vaccine Safety Signals: A Validation StudyFunding None.
Background Social media platforms have emerged as a valuable data source for public health research and surveillance. Monitoring of social media and user-generated data on the Web enables timely and inexpensive collection of information, overcoming time lag and cost of traditional health reporting systems.
Objectives This article identifies personally experienced coronavirus disease 2019 (COVID-19) vaccine reactions expressed on Twitter and validate the findings against an established vaccine reactions reporting system.
Methods We collected around 3 million tweets from 1.4 million users between February 1, 2021, to January 31, 2022, using COVID-19 vaccines and vaccine reactions keyword lists. We performed topic modeling on a sample of the data and applied a modified F1 scoring technique to identify a topic that best differentiated vaccine-related personal health mentions. We then manually annotated 4,000 of the records from this topic, which were used to train a transformer-based classifier to identify likely personally experienced vaccine reactions. Applying the trained classifier to the entire data set allowed us to select records we could use to quantify potential vaccine side effects. Adverse events following immunization (AEFI) referred to in these records were compared with those reported to the state of Victoria's spontaneous vaccine safety surveillance system, SAEFVIC (Surveillance of Adverse Events Following Vaccination In the Community).
Results The most frequently mentioned potential vaccine reactions generally aligned with SAEFVIC data. Notable exceptions were increased Twitter reporting of bleeding-related AEFI and allergic reactions, and more frequent SAEFVIC reporting of cardiac AEFI.
Conclusion Social media conversations are a potentially valuable supplementary data source for detecting vaccine adverse event mentions. Monitoring of online observations about new vaccine-related personal health experiences has the capacity to provide early warnings about emerging vaccine safety issues.
KeywordsCOVID-19 vaccines - social media - Internet - artificial intelligence - patient-generated health data - pharmacovigilance - immunization - vaccination - adverse events
Protection of Human and Animal Subjects
Ethics approval for this study was granted by The Royal Children's Hospital Melbourne Human Research Ethics Committee (HREC) (project ID 85026).
Received: 11 July 2022
Accepted: 25 October 2022
Accepted Manuscript online:
09 November 2022
Article published online:
04 January 2023
© 2023. Thieme. All rights reserved.
Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany
- 1 Black S. Recognizing the importance of vaccine confidence. EBioMedicine 2016; 12: 28-29
- 2 Jacobson RM, Adegbenro A, Pankratz VS, Poland GA. Adverse events and vaccination-the lack of power and predictability of infrequent events in pre-licensure study. Vaccine 2001; 19 (17-19): 2428-2433
- 3 Hervé C, Laupèze B, Del Giudice G, Didierlaurent AM, Tavares Da Silva F. The how's and what's of vaccine reactogenicity. NPJ Vaccines 2019; 4 (01) 39
- 4 Oliveira PMN, Lignani LK, Conceição DAD. et al. Surveillance of adverse events following immunization in the late 2010s: an overview of the importance, tools, and challenges [in Portuguese]. Cad Saude Publica 2020; 36 (Suppl. 02) e00182019
- 5 Pal SN, Duncombe C, Falzon D, Olsson S. WHO strategy for collecting safety data in public health programmes: complementing spontaneous reporting systems. Drug Saf 2013; 36 (02) 75-81
- 6 Paul MJ, Dredze M. Social monitoring for public health. Synth Lect Inf Concepts Retr Serv 2017; 9 (05) 1-183
- 7 Wilson AE, Lehmann CU, Saleh SN, Hanna J, Medford RJ. Social media: a new tool for outbreak surveillance. Antimicrob Steward Healthc Epidemiol 2021; 1 (01) e50
- 8 Qin L, Sun Q, Wang Y. et al. Prediction of number of cases of 2019 novel coronavirus (COVID-19) using social media search index. Int J Environ Res Public Health 2020; 17 (07) E2365
- 9 Medford RJ, Saleh SN, Sumarsono A, Perl TM, Lehmann CU. An “Infodemic”: leveraging high-volume twitter data to understand early public sentiment for the coronavirus disease 2019 outbreak. Open Forum Infect Dis 2020; 7 (07) ofaa258
- 10 Tsao SF, Chen H, Tisseverasinghe T, Yang Y, Li L, Butt ZA. What social media told us in the time of COVID-19: a scoping review. Lancet Digit Health 2021; 3 (03) e175-e194
- 11 Lanier HD, Diaz MI, Saleh SN, Lehmann CU, Medford RJ. Analyzing COVID-19 disinformation on Twitter using the hashtags #scamdemic and #plandemic: retrospective study. PLoS One 2022; 17 (06) e0268409
- 12 Cinelli M, Quattrociocchi W, Galeazzi A. et al. The COVID-19 social media infodemic. Sci Rep 2020; 10 (01) 16598
- 13 Sarker A, Ginn R, Nikfarjam A. et al. Utilizing social media data for pharmacovigilance: a review. J Biomed Inform 2015; 54: 202-212
- 14 Lardon J, Abdellaoui R, Bellet F. et al. Adverse drug reaction identification and extraction in social media: a scoping review. J Med Internet Res 2015; 17 (07) e171
- 15 Karafillakis E, Martin S, Simas C. et al. Methods for social media monitoring related to vaccination: systematic scoping review. JMIR Public Health Surveill 2021; 7 (02) e17149
- 16 Yin Z, Fabbri D, Rosenbloom ST, Malin B. A scalable framework to detect personal health mentions on Twitter. J Med Internet Res 2015; 17 (06) e138
- 17 Yu H, Ho C, Juan Y, Lin C. Libshorttext: a library for short-text classification and analysis. Rapport interne, Department of Computer Science, National Taiwan University Software. Published online 2013. Accessed November 25, 2022, at: http://www csie ntu edu tw/cjlin/libshorttext
- 18 Wang J, Zhao L. Multi-instance domain adaptation for vaccine adverse event detection. 2018: 97-106
- 19 Lian AT, Du J, Tang L. Using a machine learning approach to monitor COVID-19 vaccine adverse events (VAE) from Twitter data. Vaccines (Basel) 2022; 10 (01) 103
- 20 Twitter. Twitter API Academic Research. Published 2022. Accessed September 2, 2022, at: https://developer.twitter.com/en/products/twitter-api/academic-research
- 21 Pigott F, Kolb J, Montague J, Gonzales A, Moffitt J. Searchtweets, Wrapper for Twitter API v2 search endpoints. Published 2021. Accessed September 2, 2022, at: https://pypi.org/project/searchtweets-v2/
- 22 Clothier HJ, Crawford NW, Kempe A, Buttery JP. Surveillance of adverse events following immunisation: the model of SAEFVIC, Victoria. Commun Dis Intell Q Rep 2011; 35 (04) 294-298
- 23 SAEFVIC. Surveillance of adverse events following vaccination in the community. Published 2022. Accessed November 25, 2022, at: https://www.mcri.edu.au/research/research-areas/infection-and-immunity/saefvic
- 24 McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012; 22 (03) 276-282
- 25 Khademi Habibabadi S, Haghighi PD, Habibabadi SK, Haghighi PD, Khademi S, Haghighi PD. Topic modelling for identification of vaccine reactions in Twitter. ACM International Conference Proceeding Series. Published online 2019:31. Accessed November 25, 2022 at: https://dl.acm.org/doi/abs/10.1145/3290688.3290735
- 26 Řehůřek R, Sojka P. Software Framework for Topic Modelling with Large Corpora. Proceedings of LREC 2010 Workshop New Challenges. Published online 2010. Accessed November 25, 2022, at: http://www.muni.cz/research/publications/884893
- 27 Nguyen DQ. jLDADMM: A Java package for the LDA and DMM topic models. 2018 (Dmm):1–5. Accessed November 25, 2022, at: http://arxiv.org/abs/1808.03835
- 28 Mining of Twitter Conversations: 2-Phase Classification Study. JMIR Med Inform 2022; 10 (06) E34305
- 29 Nguyen DQ, Vu T, Nguyen AT. BERTweet: a pre-trained language model for English Tweets. arXiv preprint arXiv:200510200. Published online 2020
- 30 WHO. . COVID-19 Vaccines: Safety Surveillance Manual; Published online 2021. Accessed December 3, 2022 at: https://www.who.int/publications/i/item/9789240032781
- 31 Agency EM. COVID-19 Vaccine AstraZeneca: PRAC preliminary view suggests no specific issue with batch used in Austria | European Medicines Agency. Published 2021. Accessed April 29, 2022, at: https://www.ema.europa.eu/en/news/covid-19-vaccine-astrazeneca-prac-preliminary-view-suggests-no-specific-issue-batch-used-austria
- 32 Crawford NW, Clothier H, Hodgson K, Selvaraj G, Easton ML, Buttery JP. Active surveillance for adverse events following immunization. Expert Rev Vaccines 2014; 13 (02) 265-276
- 33 Danchin M, Buttery J. COVID-19 vaccine hesitancy: a unique set of challenges. Intern Med J 2021; 51 (12) 1987-1989
- 34 Niemiec E. COVID-19 and misinformation: is censorship of social media a remedy to the spread of medical misinformation?. EMBO Rep 2020; 21 (11) e51420